/**
Benchmark corpus loading.
Small datasets are pinned by `nix/packages/wired-bench-data.nix` and exposed
to the devshell as `$WIRED_BENCH_DATA`. Multi-gigabyte corpora are deliberately
not copied into every developer's Nix store: put their normalized files in
`$WIRED_BENCH_EXTERNAL_DATA` and select them explicitly with
`$WIRED_BENCH_DATASETS` (see $(MREF sparkles,wired_bench,runner)).
*/
module (package) sparklessparkles.(package) sparkles.wired_benchwired_bench.(module) sparkles.wired_bench.dataBenchmark corpus loading.
Small datasets are pinned by nix/packages/wired-bench-data.nix and exposed
to the devshell as $WIRED_BENCH_DATA. Multi-gigabyte corpora are deliberately
not copied into every developer's Nix store: put their normalized files in
$WIRED_BENCH_EXTERNAL_DATA and select them explicitly with
$WIRED_BENCH_DATASETS (see sparkles.wired_bench.runner).
data;
import (package) stdstd.(package) std.algorithmalgorithm.(module) std.algorithm.iterationThis is a submodule of std.algorithm.
It contains generic iteration algorithms.
Function Name Description cache Eagerly evaluates and caches another range's front. lazyCache Lazily evaluates and caches another range's front, unlike cache. cacheBidirectional As above, but also provides back and popBack. chunkBy chunkBy!((a,b) => a[1] == b[1])([[1, 1], [1, 2], [2, 2], [2, 1]]) returns a range containing 3 subranges: the first with just [1, 1]; the second with the elements [1, 2] and [2, 2]; and the third with just [2, 1]. cumulativeFold cumulativeFold!((a, b) => a + b)([1, 2, 3, 4]) returns a lazily-evaluated range containing the successive reduced values 1, 3, 6, 10. each each!writeln([1, 2, 3]) eagerly prints the numbers 1, 2 and 3 on their own lines. filter filter!(a => a > 0)([1, -1, 2, 0, -3]) iterates over elements 1 and 2. filterBidirectional Similar to filter, but also provides back and popBack at a small increase in cost. fold fold!((a, b) => a + b)([1, 2, 3, 4]) returns 10. group group([5, 2, 2, 3, 3]) returns a range containing the tuples tuple(5, 1), tuple(2, 2), and tuple(3, 2). joiner joiner(["hello", "world!"], "; ") returns a range that iterates over the characters "hello; world!". No new string is created - the existing inputs are iterated. map map!(a => a * 2)([1, 2, 3]) lazily returns a range with the numbers 2, 4, 6. mean Colloquially known as the average, mean([1, 2, 3]) returns 2. permutations Lazily computes all permutations using Heap's algorithm. reduce reduce!((a, b) => a + b)([1, 2, 3, 4]) returns 10. This is the old implementation of fold. splitWhen Lazily splits a range by comparing adjacent elements. splitter Lazily splits a range by a separator, element predicate or whitespace. substitute [1, 2].substitute(1, 0.1) returns [0.1, 2]. sum Same as fold, but specialized for accurate summation. uniq Iterates over the unique elements in a range, which is assumed sorted.
Source
std/algorithm/iteration.d
iteration : (alias template) sparkles.wired_bench.data.filter = std.algorithm.iteration.filter(alias predicate) if (is(typeof(unaryFun!predicate)))``filter!(predicate)(range) returns a new range containing only elements x in range for
which predicate(x) returns true.
The predicate is passed to unaryFun, and can be either a string, or
any callable that can be executed via pred(element).
filter, (alias template) sparkles.wired_bench.data.map = std.algorithm.iteration.map(fun...) if (fun.length >= 1)Implements the homonym function (also known as transform) present
in many languages of functional flavor. The call ``map!(fun)(range)
returns a range of which elements are obtained by applying fun(a)
left to right for all elements a in range. The original ranges are
not changed. Evaluation is done lazily.
map, (alias template) sparkles.wired_bench.data.splitter = std.algorithm.iteration.splitter(alias pred = "a == b", Flag keepSeparators = No.keepSeparators, Range, Separator)(Range r, Separator s) if (is(typeof(binaryFun!pred(r.front, s)) : bool) && (hasSlicing!Range && hasLength!Range || isNarrowString!Range) && (is(ElementType!Range : Separator) || !(isForwardRange!Separator && (hasLength!Separator || isNarrowString!Separator))))Lazily splits a range using an element or range as a separator.
Separator ranges can be any narrow string type or sliceable range type.
Two adjacent separators are considered to surround an empty element in
the split range. Use filter!(a => !a.empty) on the result to compress
empty elements.
The predicate is passed to binaryFun and accepts
any callable function that can be executed via pred(element, s).
Note
If splitting a string on whitespace and token compression is desired,
consider using the ``splitter(r) overload.
Constraints
The predicate pred needs to accept an element of r and the
separator s.
splitter;
import (package) stdstd.(module) std.arrayFunctions and types that manipulate built-in arrays and associative arrays.
This module provides all kinds of functions to create, manipulate or convert arrays:
Function Name Description
| array |
Returns a copy of the input in a newly allocated dynamic array.
|
| appender |
Returns a new Appender or RefAppender initialized with a given array.
|
| assocArray |
Returns a newly allocated associative array from a range/ranges of keys and values.
|
| byPair |
Construct a range iterating over an associative array by key/value tuples.
|
| insertInPlace |
Inserts into an existing array at a given position.
|
| join |
Concatenates a range of ranges into one array.
|
| minimallyInitializedArray |
Returns a new array of type T.
|
| replace |
Returns a new array with all occurrences of a certain subrange replaced.
|
| replaceFirst |
Returns a new array with the first occurrence of a certain subrange replaced.
|
| replaceInPlace |
Replaces all occurrences of a certain subrange and puts the result into a given array.
|
| replaceInto |
Replaces all occurrences of a certain subrange and puts the result into an output range.
|
| replaceLast |
Returns a new array with the last occurrence of a certain subrange replaced.
|
| replaceSlice |
Returns a new array with a given slice replaced.
|
| replicate |
Creates a new array out of several copies of an input array or range.
|
| sameHead |
Checks if the initial segments of two arrays refer to the same
place in memory.
|
| sameTail |
Checks if the final segments of two arrays refer to the same place
in memory.
|
| split |
Eagerly split a range or string into an array.
|
| staticArray |
Creates a new static array from given data.
|
| uninitializedArray |
Returns a new array of type T without initializing its elements.
|
Source
std/array.d
array : (alias template) sparkles.wired_bench.data.array = std.array.array(Range)(Range r) if (isIterable!Range && !isAutodecodableString!Range && !isInfinite!Range)Allocates an array and initializes it with copies of the elements
of range r.
Narrow strings are handled as follows:
If autodecoding is turned on (default), then they are handled as a separate overload.
If autodecoding is turned off, then this is equivalent to duplicating the array.
array;
import (package) stdstd.(module) std.exceptionThis module defines functions related to exceptions and general error
handling. It also defines functions intended to aid in unit testing.
Category Functions Assumptions assertNotThrown assertThrown assumeUnique assumeWontThrow mayPointTo Enforce doesPointTo enforce errnoEnforce Handlers collectException collectExceptionMsg ifThrown handle Other basicExceptionCtors emptyExceptionMsg ErrnoException RangePrimitive
Source
std/exception.d
Examples
Synopis
import core.stdc.stdlib : malloc, free;
import std.algorithm.comparison : equal;
import std.algorithm.iteration : map, splitter;
import std.algorithm.searching : endsWith;
import std.conv : ConvException, to;
import std.range : front, retro;
// use enforce like assert
int a = 3;
enforce(a > 2, "a needs to be higher than 2.");
// enforce can throw a custom exception
enforce!ConvException(a > 2, "a needs to be higher than 2.");
// enforce will return it's input
enum size = 42;
auto memory = enforce(malloc(size), "malloc failed")[0 .. size];
scope(exit) free(memory.ptr);
// collectException can be used to test for exceptions
Exception e = collectException("abc".to!int);
assert(e.file.endsWith("conv.d"));
// and just for the exception message
string msg = collectExceptionMsg("abc".to!int);
assert(msg == "Unexpected 'a' when converting from type string to type int");
// assertThrown can be used to assert that an exception is thrown
assertThrown!ConvException("abc".to!int);
// ifThrown can be used to provide a default value if an exception is thrown
assert("x".to!int().ifThrown(0) == 0);
// handle is a more advanced version of ifThrown for ranges
auto r = "12,1337z32,54".splitter(',').map!(a => to!int(a));
auto h = r.handle!(ConvException, RangePrimitive.front, (e, r) => 0);
assert(h.equal([12, 0, 54]));
assertThrown!ConvException(h.retro.equal([54, 0, 12]));
// basicExceptionCtors avoids the boilerplate when creating custom exceptions
static class MeaCulpa : Exception
{
mixin basicExceptionCtors;
}
e = collectException((){throw new MeaCulpa("diagnostic message");}());
assert(e.msg == "diagnostic message");
assert(e.file == __FILE__);
assert(e.line == __LINE__ - 3);
// assumeWontThrow can be used to cast throwing code into `nothrow`
void exceptionFreeCode() nothrow
{
// auto-decoding only throws if an invalid UTF char is given
assumeWontThrow("abc".front);
}
// assumeUnique can be used to cast mutable instance to an `immutable` one
// use with care
char[] str = " mutable".dup;
str[0 .. 2] = "im";
immutable res = assumeUnique(str);
assert(res == "immutable");
exception : (alias template) sparkles.wired_bench.data.enforce = std.exception.enforce(E : Throwable = Exception) if (is(typeof(new E("", string.init, size_t.init)) : Throwable) || is(typeof(new E(string.init, size_t.init)) : Throwable))Enforces that the given value is true.
If the given value is false, an exception is thrown.
The
msg - error message as a string
dg - custom delegate that return a string and is only called if an exception occurred
ex - custom exception to be thrown. It is lazy and is only created if an exception occurred
enforce;
import (package) stdstd.(module) std.fileUtilities for manipulating files and scanning directories. Functions
in this module handle files as a unit, e.g., read or write one file
at a time. For opening files and manipulating them via handles refer
to module std.stdio.
Category Functions General exists isDir isFile isSymlink rename thisExePath Directories chdir dirEntries getcwd mkdir mkdirRecurse rmdir rmdirRecurse tempDir Files append copy read readText remove slurp write Symlinks symlink readLink Attributes attrIsDir attrIsFile attrIsSymlink getAttributes getLinkAttributes getSize setAttributes Timestamp getTimes getTimesWin setTimes timeLastModified timeLastAccessed timeStatusChanged Other DirEntry FileException PreserveAttributes SpanMode getAvailableDiskSpace
Source
std/file.d
file : (alias template) sparkles.wired_bench.data.exists = std.file.exists(R)(R name) if (isSomeFiniteCharInputRange!R && !isConvertibleToString!R)Determine whether the given file (or directory) exists.
exists, (alias template) sparkles.wired_bench.data.readText = std.file.readText(S = string, R)(auto ref R name) if (isSomeString!S && (isSomeFiniteCharInputRange!R || is(StringTypeOf!R)))Reads and validates (using validate) a text file. S can be
an array of any character type. However, no width or endian conversions are
performed. So, if the width or endianness of the characters in the given
file differ from the width or endianness of the element type of S, then
validation will fail.
readText;
import (package) stdstd.(module) std.mmfileRead and write memory mapped files.
Memory mapped files are a mechanism in operating systems that allows
file access through virtual memory. After opening a file with MmFile,
the contents can be read from or written to with standard slice / pointer operations.
Changes to the memory are automatically reflected in the underlying file.
Memory mapping can increase I/O performance of large files, compared to buffered
read / write operations from std.file and std.stdio. However, I/O errors are
not handled as safely: when for example the disk that the file is on gets removed,
reading from it may result in a segfault.
References
https://en.wikipedia.org/wiki/Memory-mapped_file
Source
std/mmfile.d
mmfile : (class) std.mmfile.MmFileMmFile objects control the memory mapped file resource.
Examples
Read an existing file
import std.file;
std.file.write(deleteme, "hello"); // deleteme is a temporary filename
scope(exit) remove(deleteme);
// Use a scope class so the file will be closed at the end of this function
scope mmfile = new MmFile(deleteme);
assert(mmfile.length == "hello".length);
// Access file contents with the slice operator
// This is typed as `void[]`, so cast to `char[]` or `ubyte[]` to use it
const data = cast(const(char)[]) mmfile[];
// At this point, the file content may not have been read yet.
// In that case, the following memory access will intentionally
// trigger a page fault, causing the kernel to load the file contents
assert(data[0 .. 5] == "hello");
Write a new file
import std.file;
scope(exit) remove(deleteme);
scope mmfile = new MmFile(deleteme, MmFile.Mode.readWriteNew, 5, null);
assert(mmfile.length == 5);
auto data = cast(ubyte[]) mmfile[];
// This write to memory will be reflected in the file contents
data[] = '\n';
mmfile.flush();
assert(std.file.read(deleteme) == "\n\n\n\n\n");
MmFile;
import (package) stdstd.(module) std.pathThis module is used to manipulate path strings.
All functions, with the exception of expandTilde (and in some
cases absolutePath and relativePath), are pure
string manipulation functions; they don't depend on any state outside
the program, nor do they perform any actual file system actions.
This has the consequence that the module does not make any distinction
between a path that points to a directory and a path that points to a
file, and it does not know whether or not the object pointed to by the
path actually exists in the file system.
To differentiate between these cases, use isDir and
exists.
Note that on Windows, both the backslash (\) and the slash (/)
are in principle valid directory separators. This module treats them
both on equal footing, but in cases where a new separator is
added, a backslash will be used. Furthermore, the buildNormalizedPath
function will replace all slashes with backslashes on that platform.
In general, the functions in this module assume that the input paths
are well-formed. (That is, they should not contain invalid characters,
they should follow the file system's path format, etc.) The result
of calling a function on an ill-formed path is undefined. When there
is a chance that a path or a file name is invalid (for instance, when it
has been input by the user), it may sometimes be desirable to use the
isValidFilename and isValidPath functions to check
this.
Most functions do not perform any memory allocations, and if a string is
returned, it is usually a slice of an input string. If a function
allocates, this is explicitly mentioned in the documentation.
Category Functions Normalization absolutePath asAbsolutePath asNormalizedPath asRelativePath buildNormalizedPath buildPath chainPath expandTilde Partitioning baseName dirName dirSeparator driveName pathSeparator pathSplitter relativePath rootName stripDrive Validation isAbsolute isDirSeparator isRooted isValidFilename isValidPath Extension defaultExtension extension setExtension stripExtension withDefaultExtension withExtension Other filenameCharCmp filenameCmp globMatch CaseSensitive
Source
std/path.d
path : (alias template) sparkles.wired_bench.data.buildPath = std.path.buildPath(Range)(scope Range segments) if (isInputRange!Range && !isInfinite!Range && isSomeString!(ElementType!Range))Combines one or more path segments.
This function takes a set of path segments, given as an input
range of string elements or as a set of string arguments,
and concatenates them with each other. Directory separators
are inserted between segments if necessary. If any of the
path segments are absolute (as defined by isAbsolute), the
preceding segments will be dropped.
On Windows, if one of the path segments are rooted, but not absolute
(e.g. \foo), all preceding path segments down to the previous
root will be dropped. (See below for an example.)
This function always allocates memory to hold the resulting path.
The variadic overload is guaranteed to only perform a single
allocation, as is the range version if paths is a forward
range.
buildPath;
import (package) stdstd.(module) std.processFunctions for starting and interacting with other processes, and for
working with the current process' execution environment.
Process handling
`spawnProcess` spawns a new `process`, optionally assigning it an
arbitrary set of standard input, output, and error streams.
The function returns immediately, leaving the child process to execute
in parallel with its parent. All other functions in this module that
spawn processes are built around spawnProcess.
`wait` makes the parent `process` wait for a child `process` to
terminate. In general one should always do this, to avoid
child processes becoming "zombies" when the parent process exits.
Scope guards are perfect for this – see the spawnProcess
documentation for examples. tryWait is similar to wait,
but does not block if the process has not yet terminated.
`pipeProcess` also spawns a child `process` which runs
in parallel with its parent. However, instead of taking
arbitrary streams, it automatically creates a set of
pipes that allow the parent to communicate with the child
through the child's standard input, output, and/or error streams.
This function corresponds roughly to C's popen function.
`execute` starts a new `process` and waits for it
to complete before returning. Additionally, it captures
the process' standard output and error streams and returns
the output of these as a string.
`spawnShell`, `pipeShell` and `executeShell` work like
spawnProcess, pipeProcess and execute, respectively,
except that they take a single command string and run it through
the current user's default command interpreter.
executeShell corresponds roughly to C's system function.
`kill` attempts to terminate a running `process`.
The following table compactly summarises the different process creation
functions and how they relate to each other:
Runs program directly
Runs shell command Low-level process creation spawnProcess spawnShell Automatic input/output redirection using pipes pipeProcess pipeShell Execute and wait for completion, collect output execute executeShell
Other functionality
`pipe` is used to create unidirectional pipes.
`environment` is an interface through which the current `process`'
environment variables can be read and manipulated.
`escapeShellCommand` and `escapeShellFileName` are useful
for constructing shell command lines in a portable way.
Source
std/process.d
Note
Most of the functionality in this module is not available on iOS, tvOS
and watchOS. The only functions available on those platforms are:
environment, thisProcessID and thisThreadID.
process : (class) std.process.environmentManipulates environment variables using an associative-array-like
interface.
This class contains only static methods, and cannot be instantiated.
See below for examples of use.
environment;
import (package) stdstd.(module) std.stringString handling functions.
Category Functions Searching column indexOf indexOfAny indexOfNeither lastIndexOf lastIndexOfAny lastIndexOfNeither Comparison isNumeric Mutation capitalize Pruning and Filling center chomp chompPrefix chop detabber detab entab entabber leftJustify outdent rightJustify strip stripLeft stripRight wrap Substitution abbrev soundex soundexer succ tr translate Miscellaneous assumeUTF fromStringz lineSplitter representation splitLines toStringz Objects of types string, wstring, and dstring are value types and cannot be mutated element-by-element. For using mutation during building strings, use char[], wchar[], or dchar[]. The xxxstring types are preferable because they don't exhibit undesired aliasing, thus making code more robust.
The following functions are publicly imported:
Module Functions Publicly imported functions std.algorithm cmp, std,algorithm,comparison count, std,algorithm,searching endsWith, std,algorithm,searching startsWith, std,algorithm,searching std.array join, std,array replace, std,array replaceInPlace, std,array split, std,array empty, std,array std.format format, std,format sformat, std,format std.uni icmp, std,uni toLower, std,uni toLowerInPlace, std,uni toUpper, std,uni toUpperInPlace, std,uni There is a rich set of functions for string handling defined in other modules. Functions related to Unicode and ASCII are found in std.uni and std.ascii, respectively. Other functions that have a wider generality than just strings can be found in std.algorithm and std.range.
Source
std/string.d
string : (alias template) sparkles.wired_bench.data.strip = std.string.strip(Range)(Range str) if (isSomeString!Range || isRandomAccessRange!Range && hasLength!Range && hasSlicing!Range && !isConvertibleToString!Range && isSomeChar!(ElementEncodingType!Range))Strips both leading and trailing whitespace (as defined by
isWhite) or as specified in the second argument.
strip;
/// How one corpus is divided into JSON documents.
enum (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat
{
(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, /// one RFC 8259 JSON text
(enum value) sparkles.wired_bench.data.DatasetFormat.ndjson = 1one complete JSON text per non-empty line
ndjson, /// one complete JSON text per non-empty line
(enum value) sparkles.wired_bench.data.DatasetFormat.jsonArrayLines = 2[/] plus one comma-terminated value per line
jsonArrayLines, /// `[`/`]` plus one comma-terminated value per line
}
/// One known corpus and its on-disk convention.
struct (struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource
{
(alias) object.string = stringstring (field) string sparkles.wired_bench.data.DatasetSource.namename;
(alias) object.string = stringstring (field) string sparkles.wired_bench.data.DatasetSource.fileNamefileName;
(enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat (field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.DatasetSource.formatformat;
bool (field) bool sparkles.wired_bench.data.DatasetSource.bundledbundled;
}
/// The normal, reproducible benchmark matrix. External corpora are opt-in.
immutable (alias) object.string = stringstring[] (immutable global) immutable(string[]) sparkles.wired_bench.data.defaultDatasetNamesThe normal, reproducible benchmark matrix. External corpora are opt-in.
defaultDatasetNames =
["twitter", "citm_catalog", "canada", "github_events", "mesh",
"mesh_pretty"];
/// Every dataset name accepted by `$WIRED_BENCH_DATASETS`.
immutable (struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource[] (immutable global) immutable(sparkles.wired_bench.data.DatasetSource[]) sparkles.wired_bench.data.datasetSourcesEvery dataset name accepted by $WIRED_BENCH_DATASETS.
datasetSources =
[
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("twitter", "twitter.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, true),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("citm_catalog", "citm_catalog.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, true),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("canada", "canada.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, true),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("github_events", "github_events.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, true),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("mesh", "mesh.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, true),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("mesh_pretty", "mesh.pretty.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, true),
// The Wikidata dump is a single array on disk, but its documented physical
// layout is one entity per line. Treating those lines as records keeps the
// live parse tree bounded under sustained runs. `recordLine` removes the
// surrounding array and each line's separator comma.
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("wikidata", "wikidata.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.jsonArrayLines = 2[/] plus one comma-terminated value per line
jsonArrayLines, false),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("osm", "osm.json", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document, false),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("cloudtrail", "cloudtrail.ndjson", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.ndjson = 1one complete JSON text per non-empty line
ndjson, false),
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource("elasticsearch", "elasticsearch.ndjson", (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.ndjson = 1one complete JSON text per non-empty line
ndjson, false),
];
/// One loaded benchmark corpus.
struct (struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset
{
(alias) object.string = stringstring (field) string sparkles.wired_bench.data.Dataset.namedataset name, e.g. twitter
name; /// dataset name, e.g. `twitter`
const(char)[] (field) const(char)[] sparkles.wired_bench.data.Dataset.textthe raw corpus text
text; /// the raw corpus text
(enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat (field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.Dataset.formatdocument framing
format; /// document framing
private (class) std.mmfile.MmFileMmFile objects control the memory mapped file resource.
Examples
Read an existing file
import std.file;
std.file.write(deleteme, "hello"); // deleteme is a temporary filename
scope(exit) remove(deleteme);
// Use a scope class so the file will be closed at the end of this function
scope mmfile = new MmFile(deleteme);
assert(mmfile.length == "hello".length);
// Access file contents with the slice operator
// This is typed as `void[]`, so cast to `char[]` or `ubyte[]` to use it
const data = cast(const(char)[]) mmfile[];
// At this point, the file content may not have been read yet.
// In that case, the following memory access will intentionally
// trigger a page fault, causing the kernel to load the file contents
assert(data[0 .. 5] == "hello");
Write a new file
import std.file;
scope(exit) remove(deleteme);
scope mmfile = new MmFile(deleteme, MmFile.Mode.readWriteNew, 5, null);
assert(mmfile.length == 5);
auto data = cast(ubyte[]) mmfile[];
// This write to memory will be reflected in the file contents
data[] = '\n';
mmfile.flush();
assert(std.file.read(deleteme) == "\n\n\n\n\n");
MmFile (field) std.mmfile.MmFile sparkles.wired_bench.data.Dataset.mappingkeeps an external corpus's read-only map alive
mapping; /// keeps an external corpus's read-only map alive
/// A lazy range of JSON texts in a line-oriented corpus.
auto sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)) sparkles.wired_bench.data.Dataset.records() const pure nothrow @safeA lazy range of JSON texts in a line-oriented corpus.
records() const @safe
in ((field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.Dataset.formatdocument framing
format != (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.document = 0one RFC 8259 JSON text
document)
{
const (local variable) const(sparkles.wired_bench.data.DatasetFormat) framingframing = (field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.Dataset.formatdocument framing
format;
return (field) const(char)[] sparkles.wired_bench.data.Dataset.textthe raw corpus text
text.std.algorithm.iteration.splitter!("a == b", Flag.no, const(char)[], char).Result std.algorithm.iteration.splitter!("a == b", Flag.no, const(char)[], char)(const(char)[] r, char s) pure nothrow @nogc @safeLazily splits a range using an element or range as a separator.
Separator ranges can be any narrow string type or sliceable range type.
Two adjacent separators are considered to surround an empty element in
the split range. Use filter!(a => !a.empty) on the result to compress
empty elements.
The predicate is passed to binaryFun and accepts
any callable function that can be executed via pred(element, s).
Note
If splitting a string on whitespace and token compression is desired,
consider using the ``splitter(r) overload.
Constraints
The predicate pred needs to accept an element of r and the
separator s.
Examples
Basic splitting with characters and numbers.
import std.algorithm.comparison : equal;
assert("a|bc|def".splitter('|').equal([ "a", "bc", "def" ]));
int[] a = [1, 0, 2, 3, 0, 4, 5, 6];
int[][] w = [ [1], [2, 3], [4, 5, 6] ];
assert(a.splitter(0).equal(w));
Basic splitting with characters and numbers and keeping sentinels.
import std.algorithm.comparison : equal;
import std.typecons : Yes;
assert("a|bc|def".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "a", "|", "bc", "|", "def" ]));
int[] a = [1, 0, 2, 3, 0, 4, 5, 6];
int[][] w = [ [1], [0], [2, 3], [0], [4, 5, 6] ];
assert(a.splitter!("a == b", Yes.keepSeparators)(0).equal(w));
Adjacent separators.
import std.algorithm.comparison : equal;
assert("|ab|".splitter('|').equal([ "", "ab", "" ]));
assert("ab".splitter('|').equal([ "ab" ]));
assert("a|b||c".splitter('|').equal([ "a", "b", "", "c" ]));
assert("hello world".splitter(' ').equal([ "hello", "", "world" ]));
auto a = [ 1, 2, 0, 0, 3, 0, 4, 5, 0 ];
auto w = [ [1, 2], [], [3], [4, 5], [] ];
assert(a.splitter(0).equal(w));
Adjacent separators and keeping sentinels.
import std.algorithm.comparison : equal;
import std.typecons : Yes;
assert("|ab|".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "", "|", "ab", "|", "" ]));
assert("ab".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "ab" ]));
assert("a|b||c".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "a", "|", "b", "|", "", "|", "c" ]));
assert("hello world".splitter!("a == b", Yes.keepSeparators)(' ')
.equal([ "hello", " ", "", " ", "world" ]));
auto a = [ 1, 2, 0, 0, 3, 0, 4, 5, 0 ];
auto w = [ [1, 2], [0], [], [0], [3], [0], [4, 5], [0], [] ];
assert(a.splitter!("a == b", Yes.keepSeparators)(0).equal(w));
Empty and separator-only ranges.
import std.algorithm.comparison : equal;
import std.range : empty;
assert("".splitter('|').empty);
assert("|".splitter('|').equal([ "", "" ]));
assert("||".splitter('|').equal([ "", "", "" ]));
Empty and separator-only ranges and keeping sentinels.
import std.algorithm.comparison : equal;
import std.typecons : Yes;
import std.range : empty;
assert("".splitter!("a == b", Yes.keepSeparators)('|').empty);
assert("|".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "", "|", "" ]));
assert("||".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "", "|", "", "|", "" ]));
Use a range for splitting
import std.algorithm.comparison : equal;
assert("a=>bc=>def".splitter("=>").equal([ "a", "bc", "def" ]));
assert("a|b||c".splitter("||").equal([ "a|b", "c" ]));
assert("hello world".splitter(" ").equal([ "hello", "world" ]));
int[] a = [ 1, 2, 0, 0, 3, 0, 4, 5, 0 ];
int[][] w = [ [1, 2], [3, 0, 4, 5, 0] ];
assert(a.splitter([0, 0]).equal(w));
a = [ 0, 0 ];
assert(a.splitter([0, 0]).equal([ (int[]).init, (int[]).init ]));
a = [ 0, 0, 1 ];
assert(a.splitter([0, 0]).equal([ [], [1] ]));
Use a range for splitting
import std.algorithm.comparison : equal;
import std.typecons : Yes;
assert("a=>bc=>def".splitter!("a == b", Yes.keepSeparators)("=>")
.equal([ "a", "=>", "bc", "=>", "def" ]));
assert("a|b||c".splitter!("a == b", Yes.keepSeparators)("||")
.equal([ "a|b", "||", "c" ]));
assert("hello world".splitter!("a == b", Yes.keepSeparators)(" ")
.equal([ "hello", " ", "world" ]));
int[] a = [ 1, 2, 0, 0, 3, 0, 4, 5, 0 ];
int[][] w = [ [1, 2], [0, 0], [3, 0, 4, 5, 0] ];
assert(a.splitter!("a == b", Yes.keepSeparators)([0, 0]).equal(w));
a = [ 0, 0 ];
assert(a.splitter!("a == b", Yes.keepSeparators)([0, 0])
.equal([ (int[]).init, [0, 0], (int[]).init ]));
a = [ 0, 0, 1 ];
assert(a.splitter!("a == b", Yes.keepSeparators)([0, 0])
.equal([ [], [0, 0], [1] ]));
Custom predicate functions.
import std.algorithm.comparison : equal;
import std.ascii : toLower;
assert("abXcdxef".splitter!"a.toLower == b"('x').equal(
[ "ab", "cd", "ef" ]));
auto w = [ [0], [1], [2] ];
assert(w.splitter!"a.front == b"(1).equal([ [[0]], [[2]] ]));
Custom predicate functions.
import std.algorithm.comparison : equal;
import std.typecons : Yes;
import std.ascii : toLower;
assert("abXcdxef".splitter!("a.toLower == b", Yes.keepSeparators)('x')
.equal([ "ab", "X", "cd", "x", "ef" ]));
auto w = [ [0], [1], [2] ];
assert(w.splitter!("a.front == b", Yes.keepSeparators)(1)
.equal([ [[0]], [[1]], [[2]] ]));
Leading separators, trailing separators, or no separators.
import std.algorithm.comparison : equal;
assert("|ab|".splitter('|').equal([ "", "ab", "" ]));
assert("ab".splitter('|').equal([ "ab" ]));
Leading separators, trailing separators, or no separators.
import std.algorithm.comparison : equal;
import std.typecons : Yes;
assert("|ab|".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "", "|", "ab", "|", "" ]));
assert("ab".splitter!("a == b", Yes.keepSeparators)('|')
.equal([ "ab" ]));
Splitter returns bidirectional ranges if the delimiter is a single element
import std.algorithm.comparison : equal;
import std.range : retro;
assert("a|bc|def".splitter('|').retro.equal([ "def", "bc", "a" ]));
Splitter returns bidirectional ranges if the delimiter is a single element
import std.algorithm.comparison : equal;
import std.typecons : Yes;
import std.range : retro;
assert("a|bc|def".splitter!("a == b", Yes.keepSeparators)('|')
.retro.equal([ "def", "|", "bc", "|", "a" ]));
splitter('\n')
.sparkles.wired_bench.data.Dataset.records.MapResult!(__lambda_L77_C19, Result) sparkles.wired_bench.data.Dataset.records.map!(std.algorithm.iteration.splitter!("a == b", Flag.no, const(char)[], char).Result)(std.algorithm.iteration.splitter!("a == b", Flag.no, const(char)[], char).Result r) pure nothrow @nogc @safeImplements the homonym function (also known as transform) present
in many languages of functional flavor. The call ``map!(fun)(range)
returns a range of which elements are obtained by applying fun(a)
left to right for all elements a in range. The original ranges are
not changed. Evaluation is done lazily.
Examples
import std.algorithm.comparison : equal;
import std.range : chain, only;
auto squares =
chain(only(1, 2, 3, 4), only(5, 6)).map!(a => a * a);
assert(equal(squares, only(1, 4, 9, 16, 25, 36)));
Multiple functions can be passed to map. In that case, the
element type of map is a tuple containing one element for each
function.
auto sums = [2, 4, 6, 8];
auto products = [1, 4, 9, 16];
size_t i = 0;
foreach (result; [ 1, 2, 3, 4 ].map!("a + a", "a * a"))
{
assert(result[0] == sums[i]);
assert(result[1] == products[i]);
++i;
}
You may alias map with some function(s) to a symbol and use
it separately:
import std.algorithm.comparison : equal;
import std.conv : to;
alias stringize = map!(to!string);
assert(equal(stringize([ 1, 2, 3, 4 ]), [ "1", "2", "3", "4" ]));
map!(line => recordLine(line, framing))
.sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)) sparkles.wired_bench.data.Dataset.records.filter!(sparkles.wired_bench.data.Dataset.records.MapResult!(__lambda_L77_C19, Result))(sparkles.wired_bench.data.Dataset.records.MapResult!(__lambda_L77_C19, Result) range) pure nothrow @nogc @safefilter`!(predicate)(`range`)` returns a new `range` containing only elements `x` in range`` for
which predicate(x) returns true.
The predicate is passed to unaryFun, and can be either a string, or
any callable that can be executed via pred(element).
Examples
import std.algorithm.comparison : equal;
import std.math.operations : isClose;
import std.range;
int[] arr = [ 1, 2, 3, 4, 5 ];
// Filter below 3
auto small = filter!(a => a < 3)(arr);
assert(equal(small, [ 1, 2 ]));
// Filter again, but with Uniform Function Call Syntax (UFCS)
auto sum = arr.filter!(a => a < 3);
assert(equal(sum, [ 1, 2 ]));
// In combination with chain() to span multiple ranges
int[] a = [ 3, -2, 400 ];
int[] b = [ 100, -101, 102 ];
auto r = chain(a, b).filter!(a => a > 0);
assert(equal(r, [ 3, 400, 100, 102 ]));
// Mixing convertible types is fair game, too
double[] c = [ 2.5, 3.0 ];
auto r1 = chain(c, a, b).filter!(a => cast(int) a != a);
assert(isClose(r1, [ 2.5 ]));
filter!(line => line.length);
}
}
/// The corpus directory: the explicit value if given, else `$WIRED_BENCH_DATA`.
(alias) object.string = stringstring string sparkles.wired_bench.data.resolveDataDir(string explicitDir) @safeThe corpus directory: the explicit value if given, else $WIRED_BENCH_DATA.
resolveDataDir((alias) object.string = stringstring (parameter) string explicitDirexplicitDir) @safe
{
if ((parameter) string explicitDirexplicitDir.(field) ulong string.lengthlength)
return (parameter) string explicitDirexplicitDir;
const (local variable) const(string) envenv = (class) std.process.environmentManipulates environment variables using an associative-array-like
interface.
This class contains only static methods, and cannot be instantiated.
See below for examples of use.
environment.string std.process.environment.get(scope const(char)[] name, string defaultValue = null) @safeRetrieves the value of the environment variable with the given name,
or a default value if the variable doesn't exist.
Unlike environment.opIndex, this function never throws on Posix.
auto sh = environment.get("SHELL", "/bin/sh");
This function is also useful in checking for the existence of an
environment variable.
auto myVar = environment.get("MYVAR");
if (myVar is null)
{
// Environment variable doesn't exist.
// Note that we have to use 'is' for the comparison, since
// myVar == null is also true if the variable exists but is
// empty.
}
get("WIRED_BENCH_DATA");
bool std.exception.enforce!().enforce!bool(bool value, lazy const(char)[] msg = null, string file = __FILE__, ulong line = cast(ulong)__LINE__) pure @safeEnforces that the given value is true.
If the given value is false, an exception is thrown.
The
msg - error message as a string
dg - custom delegate that return a string and is only called if an exception occurred
ex - custom exception to be thrown. It is lazy and is only created if an exception occurred
enforce((local variable) const(string) envenv !is null && (local variable) const(string) envenv.(field) ulong const(string).lengthlength,
"no data directory: export WIRED_BENCH_DATA to point at the benchmark "
~ "corpora (the devshell sets it)");
return (local variable) const(string) envenv;
}
/// The external-corpus directory: explicit value, else the environment.
(alias) object.string = stringstring string sparkles.wired_bench.data.resolveExternalDataDir(string explicitDir) @safeThe external-corpus directory: explicit value, else the environment.
resolveExternalDataDir((alias) object.string = stringstring (parameter) string explicitDirexplicitDir) @safe
{
if ((parameter) string explicitDirexplicitDir.(field) ulong string.lengthlength)
return (parameter) string explicitDirexplicitDir;
return (class) std.process.environmentManipulates environment variables using an associative-array-like
interface.
This class contains only static methods, and cannot be instantiated.
See below for examples of use.
environment.string std.process.environment.get(scope const(char)[] name, string defaultValue = null) @safeRetrieves the value of the environment variable with the given name,
or a default value if the variable doesn't exist.
Unlike environment.opIndex, this function never throws on Posix.
auto sh = environment.get("SHELL", "/bin/sh");
This function is also useful in checking for the existence of an
environment variable.
auto myVar = environment.get("MYVAR");
if (myVar is null)
{
// Environment variable doesn't exist.
// Note that we have to use 'is' for the comparison, since
// myVar == null is also true if the variable exists but is
// empty.
}
get("WIRED_BENCH_EXTERNAL_DATA", "");
}
/// Looks up one catalog entry, rejecting typos before touching the filesystem.
(struct) sparkles.wired_bench.data.DatasetSourceOne known corpus and its on-disk convention.
DatasetSource sparkles.wired_bench.data.DatasetSource sparkles.wired_bench.data.datasetSource(scope const(char)[] name) @safeLooks up one catalog entry, rejecting typos before touching the filesystem.
datasetSource(scope const(char)[] (parameter) const(char)[] namename) @safe
{
foreach ((parameter) immutable(sparkles.wired_bench.data.DatasetSource) sourcesource; (immutable global) immutable(sparkles.wired_bench.data.DatasetSource[]) sparkles.wired_bench.data.datasetSourcesEvery dataset name accepted by $WIRED_BENCH_DATASETS.
datasetSources)
if ((local variable) immutable(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) string sparkles.wired_bench.data.DatasetSource.namename == (parameter) const(char)[] namename)
return (local variable) immutable(sparkles.wired_bench.data.DatasetSource) sourcesource;
bool std.exception.enforce!().enforce!bool(bool value, lazy const(char)[] msg = null, string file = __FILE__, ulong line = cast(ulong)__LINE__) pure @safeEnforces that the given value is true.
If the given value is false, an exception is thrown.
The
msg - error message as a string
dg - custom delegate that return a string and is only called if an exception occurred
ex - custom exception to be thrown. It is lazy and is only created if an exception occurred
enforce(false, "unknown dataset '" ~ (parameter) const(char)[] namename ~ "' (known: twitter, "
~ "citm_catalog, canada, github_events, mesh, mesh_pretty, wikidata, "
~ "osm, cloudtrail, elasticsearch)");
assert(false);
}
/// Loads the selected catalog entries from their bundled or external root.
(struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset[] sparkles.wired_bench.data.Dataset[] sparkles.wired_bench.data.loadDatasets(const(string[]) names, string dataDir, string externalDataDir = null) @safeLoads the selected catalog entries from their bundled or external root.
loadDatasets(const (alias) object.string = stringstring[] (parameter) const(string[]) namesnames, (alias) object.string = stringstring (parameter) string dataDirdataDir,
(alias) object.string = stringstring (parameter) string externalDataDirexternalDataDir = null) @safe
{
(struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset[] (local variable) sparkles.wired_bench.data.Dataset[] resultresult;
(local variable) sparkles.wired_bench.data.Dataset[] resultresult.ulong object.reserve!(sparkles.wired_bench.data.Dataset)(ref sparkles.wired_bench.data.Dataset[] arr, ulong newcapacity) pure nothrow @trustedReserves capacity for a slice. The capacity is the size
that the slice can grow to before the underlying array must be
reallocated or extended.
Examples
//Static array slice: no capacity. Reserve relocates.
int[4] sarray = [1, 2, 3, 4];
int[] slice = sarray[];
auto u = slice.reserve(8);
assert(u >= 8);
assert(&sarray[0] !is &slice[0]);
assert(slice.capacity == u);
//Dynamic array slices
int[] a = [1, 2, 3, 4];
a.reserve(8); //prepare a for appending 4 more items
auto p = &a[0];
u = a.capacity;
a ~= [5, 6, 7, 8];
assert(p == &a[0]); //a should not have been reallocated
assert(u == a.capacity); //a should not have been extended
reserve((parameter) const(string[]) namesnames.(field) ulong const(string[]).lengthlength);
foreach ((parameter) const(string) namename; (parameter) const(string[]) namesnames)
{
const (local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource = sparkles.wired_bench.data.DatasetSource sparkles.wired_bench.data.datasetSource(scope const(char)[] name) @safeLooks up one catalog entry, rejecting typos before touching the filesystem.
datasetSource((local variable) const(string) namename);
const (local variable) const(string) rootroot = (local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) bool sparkles.wired_bench.data.DatasetSource.bundledbundled
? (parameter) string dataDirdataDir
: string sparkles.wired_bench.data.resolveExternalDataDir(string explicitDir) @safeThe external-corpus directory: explicit value, else the environment.
resolveExternalDataDir((parameter) string externalDataDirexternalDataDir);
if ((local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) bool sparkles.wired_bench.data.DatasetSource.bundledbundled)
ulong std.exception.enforce!().enforce!ulong(ulong value, lazy const(char)[] msg = null, string file = __FILE__, ulong line = cast(ulong)__LINE__) pure @safeEnforces that the given value is true.
If the given value is false, an exception is thrown.
The
msg - error message as a string
dg - custom delegate that return a string and is only called if an exception occurred
ex - custom exception to be thrown. It is lazy and is only created if an exception occurred
enforce((local variable) const(string) rootroot.(field) ulong const(string).lengthlength, "dataset '" ~ (local variable) const(string) namename
~ "' is bundled: export WIRED_BENCH_DATA");
else
ulong std.exception.enforce!().enforce!ulong(ulong value, lazy const(char)[] msg = null, string file = __FILE__, ulong line = cast(ulong)__LINE__) pure @safeEnforces that the given value is true.
If the given value is false, an exception is thrown.
The
msg - error message as a string
dg - custom delegate that return a string and is only called if an exception occurred
ex - custom exception to be thrown. It is lazy and is only created if an exception occurred
enforce((local variable) const(string) rootroot.(field) ulong const(string).lengthlength, "dataset '" ~ (local variable) const(string) namename ~ "' is external: export "
~ "WIRED_BENCH_EXTERNAL_DATA to the directory containing "
~ (local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) string sparkles.wired_bench.data.DatasetSource.fileNamefileName);
const (local variable) const(string) pathpath = (local variable) const(string) rootroot.string std.path.buildPath!char(const(char)[][] paths...) pure nothrow @safeCombines one or more path segments.
This function takes a set of path segments, given as an input
range of string elements or as a set of string arguments,
and concatenates them with each other. Directory separators
are inserted between segments if necessary. If any of the
path segments are absolute (as defined by isAbsolute), the
preceding segments will be dropped.
On Windows, if one of the path segments are rooted, but not absolute
(e.g. \foo), all preceding path segments down to the previous
root will be dropped. (See below for an example.)
This function always allocates memory to hold the resulting path.
The variadic overload is guaranteed to only perform a single
allocation, as is the range version if paths is a forward
range.
Examples
version (Posix)
{
assert(buildPath("foo", "bar", "baz") == "foo/bar/baz");
assert(buildPath("/foo/", "bar/baz") == "/foo/bar/baz");
assert(buildPath("/foo", "/bar") == "/bar");
}
version (Windows)
{
assert(buildPath("foo", "bar", "baz") == `foo\bar\baz`);
assert(buildPath(`c:\foo`, `bar\baz`) == `c:\foo\bar\baz`);
assert(buildPath("foo", `d:\bar`) == `d:\bar`);
assert(buildPath("foo", `\bar`) == `\bar`);
assert(buildPath(`c:\foo`, `\bar`) == `c:\bar`);
}
buildPath((local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) string sparkles.wired_bench.data.DatasetSource.fileNamefileName);
bool std.exception.enforce!().enforce!bool(bool value, lazy const(char)[] msg = null, string file = __FILE__, ulong line = cast(ulong)__LINE__) pure @safeEnforces that the given value is true.
If the given value is false, an exception is thrown.
The
msg - error message as a string
dg - custom delegate that return a string and is only called if an exception occurred
ex - custom exception to be thrown. It is lazy and is only created if an exception occurred
enforce((local variable) const(string) pathpath.bool std.file.exists!string(string name) nothrow @nogc @safeDetermine whether the given file (or directory) exists.
exists, "dataset not found: " ~ (local variable) const(string) pathpath);
(local variable) sparkles.wired_bench.data.Dataset[] resultresult ~= (local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) bool sparkles.wired_bench.data.DatasetSource.bundledbundled
? (struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset((local variable) const(string) namename, string std.file.readText!(string, const(string))(ref const(string) name) @safeReads and validates (using validate) a text file. S can be
an array of any character type. However, no width or endian conversions are
performed. So, if the width or endianness of the characters in the given
file differ from the width or endianness of the element type of S, then
validation will fail.
Examples
Read file with UTF-8 text.
write(deleteme, "abc"); // deleteme is the name of a temporary file
scope(exit) remove(deleteme);
string content = readText(deleteme);
assert(content == "abc");
readText((local variable) const(string) pathpath), (local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.DatasetSource.formatformat)
: sparkles.wired_bench.data.Dataset sparkles.wired_bench.data.mapDataset(string name, string path, sparkles.wired_bench.data.DatasetFormat format) @trustedMaps a potentially enormous external corpus without a heap-sized input copy.
mapDataset((local variable) const(string) namename, (local variable) const(string) pathpath, (local variable) const(sparkles.wired_bench.data.DatasetSource) sourcesource.(field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.DatasetSource.formatformat);
}
return (local variable) sparkles.wired_bench.data.Dataset[] resultresult;
}
/// Maps a potentially enormous external corpus without a heap-sized input copy.
private (struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset sparkles.wired_bench.data.Dataset sparkles.wired_bench.data.mapDataset(string name, string path, sparkles.wired_bench.data.DatasetFormat format) @trustedMaps a potentially enormous external corpus without a heap-sized input copy.
mapDataset((alias) object.string = stringstring (parameter) string namename, (alias) object.string = stringstring (parameter) string pathpath, (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat (parameter) sparkles.wired_bench.data.DatasetFormat formatformat)
@trusted
{
auto (local variable) std.mmfile.MmFile mappingmapping = new (class) std.mmfile.MmFileMmFile objects control the memory mapped file resource.
Examples
Read an existing file
import std.file;
std.file.write(deleteme, "hello"); // deleteme is a temporary filename
scope(exit) remove(deleteme);
// Use a scope class so the file will be closed at the end of this function
scope mmfile = new MmFile(deleteme);
assert(mmfile.length == "hello".length);
// Access file contents with the slice operator
// This is typed as `void[]`, so cast to `char[]` or `ubyte[]` to use it
const data = cast(const(char)[]) mmfile[];
// At this point, the file content may not have been read yet.
// In that case, the following memory access will intentionally
// trigger a page fault, causing the kernel to load the file contents
assert(data[0 .. 5] == "hello");
Write a new file
import std.file;
scope(exit) remove(deleteme);
scope mmfile = new MmFile(deleteme, MmFile.Mode.readWriteNew, 5, null);
assert(mmfile.length == 5);
auto data = cast(ubyte[]) mmfile[];
// This write to memory will be reflected in the file contents
data[] = '\n';
mmfile.flush();
assert(std.file.read(deleteme) == "\n\n\n\n\n");
MmFile((parameter) string pathpath);
return (struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset((parameter) string namename, cast(const(char)[]) (local variable) std.mmfile.MmFile mappingmapping[], (parameter) sparkles.wired_bench.data.DatasetFormat formatformat, (local variable) std.mmfile.MmFile mappingmapping);
}
/// Normalizes one physical line into a JSON record view.
private const(char)[] const(char)[] sparkles.wired_bench.data.recordLine(return scope const(char)[] raw, sparkles.wired_bench.data.DatasetFormat format) pure nothrow @safeNormalizes one physical line into a JSON record view.
recordLine(return scope const(char)[] (parameter) const(char)[] rawraw,
(enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat (parameter) sparkles.wired_bench.data.DatasetFormat formatformat) @safe pure nothrow
{
auto (local variable) const(char)[] lineline = (parameter) const(char)[] rawraw.const(char)[] std.string.strip!(const(char)[])(const(char)[] str) pure nothrow @nogc @safeStrips both leading and trailing whitespace (as defined by
isWhite) or as specified in the second argument.
Examples
import std.uni : lineSep, paraSep;
assert(strip(" hello world ") ==
"hello world");
assert(strip("\n\t\v\rhello world\n\t\v\r") ==
"hello world");
assert(strip("hello world") ==
"hello world");
assert(strip([lineSep] ~ "hello world" ~ [lineSep]) ==
"hello world");
assert(strip([paraSep] ~ "hello world" ~ [paraSep]) ==
"hello world");
strip;
if ((parameter) sparkles.wired_bench.data.DatasetFormat formatformat == (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.jsonArrayLines = 2[/] plus one comma-terminated value per line
jsonArrayLines)
{
if ((local variable) const(char)[] lineline == "[" || (local variable) const(char)[] lineline == "]")
return null;
if ((local variable) const(char)[] lineline.(field) ulong const(char)[].lengthlength && (local variable) const(char)[] lineline[$ - 1] == ',')
(local variable) const(char)[] lineline = (local variable) const(char)[] lineline[0 .. $ - 1].const(char)[] std.string.strip!(const(char)[])(const(char)[] str) pure nothrow @nogc @safeStrips both leading and trailing whitespace (as defined by
isWhite) or as specified in the second argument.
Examples
import std.uni : lineSep, paraSep;
assert(strip(" hello world ") ==
"hello world");
assert(strip("\n\t\v\rhello world\n\t\v\r") ==
"hello world");
assert(strip("hello world") ==
"hello world");
assert(strip([lineSep] ~ "hello world" ~ [lineSep]) ==
"hello world");
assert(strip([paraSep] ~ "hello world" ~ [paraSep]) ==
"hello world");
strip;
}
return (local variable) const(char)[] lineline;
}
@("data.resolveDataDir.cliWins")
@safe unittest
{
assert(string sparkles.wired_bench.data.resolveDataDir(string explicitDir) @safeThe corpus directory: the explicit value if given, else $WIRED_BENCH_DATA.
resolveDataDir("/some/dir") == "/some/dir");
}
@("data.catalog.formatsAndDefaults")
@safe unittest
{
assert(sparkles.wired_bench.data.DatasetSource sparkles.wired_bench.data.datasetSource(scope const(char)[] name) @safeLooks up one catalog entry, rejecting typos before touching the filesystem.
datasetSource("mesh_pretty").(field) string sparkles.wired_bench.data.DatasetSource.fileNamefileName == "mesh.pretty.json");
assert(sparkles.wired_bench.data.DatasetSource sparkles.wired_bench.data.datasetSource(scope const(char)[] name) @safeLooks up one catalog entry, rejecting typos before touching the filesystem.
datasetSource("wikidata").(field) sparkles.wired_bench.data.DatasetFormat sparkles.wired_bench.data.DatasetSource.formatformat == (enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.jsonArrayLines = 2[/] plus one comma-terminated value per line
jsonArrayLines);
assert(sparkles.wired_bench.data.DatasetSource sparkles.wired_bench.data.datasetSource(scope const(char)[] name) @safeLooks up one catalog entry, rejecting typos before touching the filesystem.
datasetSource("wikidata").(field) bool sparkles.wired_bench.data.DatasetSource.bundledbundled == false);
assert((immutable global) immutable(string[]) sparkles.wired_bench.data.defaultDatasetNamesThe normal, reproducible benchmark matrix. External corpora are opt-in.
defaultDatasetNames.(field) ulong immutable(string[]).lengthlength == 6);
}
@("data.records.ndjsonAndWikidataArray")
@safe unittest
{
auto (local variable) sparkles.wired_bench.data.Dataset ndjsonndjson = (struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset("logs", " {\"a\":1}\n\n{\"b\":2}\r\n",
(enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.ndjson = 1one complete JSON text per non-empty line
ndjson);
assert((local variable) sparkles.wired_bench.data.Dataset ndjsonndjson.sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)) sparkles.wired_bench.data.Dataset.records() const pure nothrow @safeA lazy range of JSON texts in a line-oriented corpus.
records.const(char)[][] std.array.array!(sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)))(sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)) r) pure @safeAllocates an array and initializes it with copies of the elements
of range r.
Narrow strings are handled as follows:
If autodecoding is turned on (default), then they are handled as a separate overload.
If autodecoding is turned off, then this is equivalent to duplicating the array.
array == [`{"a":1}`, `{"b":2}`]);
auto (local variable) sparkles.wired_bench.data.Dataset wikidatawikidata = (struct) sparkles.wired_bench.data.DatasetOne loaded benchmark corpus.
Dataset("wikidata",
"[\n{\"id\":\"Q1\"},\n {\"id\":\"Q2\"}\n]\n",
(enum) sparkles.wired_bench.data.DatasetFormatHow one corpus is divided into JSON documents.
DatasetFormat.(enum value) sparkles.wired_bench.data.DatasetFormat.jsonArrayLines = 2[/] plus one comma-terminated value per line
jsonArrayLines);
assert((local variable) sparkles.wired_bench.data.Dataset wikidatawikidata.sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)) sparkles.wired_bench.data.Dataset.records() const pure nothrow @safeA lazy range of JSON texts in a line-oriented corpus.
records.const(char)[][] std.array.array!(sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)))(sparkles.wired_bench.data.Dataset.records.FilterResult!(__lambda_L78_C22, MapResult!(__lambda_L77_C19, Result)) r) pure @safeAllocates an array and initializes it with copies of the elements
of range r.
Narrow strings are handled as follows:
If autodecoding is turned on (default), then they are handled as a separate overload.
If autodecoding is turned off, then this is equivalent to duplicating the array.
array ==
[`{"id":"Q1"}`, `{"id":"Q2"}`]);
}