Massive speed improvement. The index generation is really fast,
and it appears that even worker numbers way higher than the core/thread
count still increase the throughput slightly.
The only missing part is the info output. That's broken, currently.
While the approach to generate and store a list of strings does allow
for this list to be re-used in the future, the I/O involved turned out
to be quite costly.
While the generation can run at up to 500 MiB/s, even compressing that
on the fly doesn't reach fast enough write speeds on a HDD.
And compression is also necessary to store this amount of data
(generation reached two TB of raw data with a word length of just three,
which is still 600 GB compressed).
But compression also makes working with that data a lot harder.
So this instead combines both the generation and search into a single
step. The intermediate result of the generation is therefore lost,
but the overall pipeline is much faster.
It seems that the simple `println!()` is really bad when the goal
is to write a lot of data to stdout.
Presumably because it's unbuffered, but also because it required the
preceding code to do a lot of allocations.
This was replaced with a buffered writer on stdout, as well as an extra
`Vec<u8>` that I can write everything to directly from the word and
delimiter iterators, without allocating a single new structure.
lint/clippy Checking for common mistakes and opportunities for code improvement
build/msvc Build for the target platform: msvc
build/linux Build for the target platform: linux
With `.enumerate()` after `.filter()`, the resulting indices didn't
properly map back to the overall mod list anymore. But the checks
afterwards relied on that.
Moving the `.enumerate()` before the `.filter()` makes sure that the
indices are correct.
This obsoletes `shlex`. The quoting turned out unnecessary, and the
splitting supported a lot more than we need. It also forced
unncessary allocations: The splitting doesn't add any characters and
keeps UTF-8 intact, so returning slices from the input is perfectly
possible.
Though this particular implementation will only come to use in the
future, as `CmdLine` still requires that the slices are cloned.
Still, the custom implementation performs about 3x faster.
lint/clippy Checking for common mistakes and opportunities for code improvement
build/msvc Build for the target platform: msvc
build/linux Build for the target platform: linux
This provides forward compatibility, in case I ever want to use those
fields. If I only added them at the time when they are needed, I would
need to come up with a process to load the `nexus.sjson` with missing
fields.
Closes#130.
lint/clippy Checking for common mistakes and opportunities for code improvement
build/msvc Build for the target platform: msvc
build/linux Build for the target platform: linux
* master:
dtmm: Fix writing Nexus image to disk
dtmm: Fix importing from `.mod` file
ci: Combine Dockerfiles into multi-stage build
ci: Rework MSVC image building
ci: Fix base pipeline
ci: Improve caching setup for image building
ci: Create build artifacts for commits on master
ci: Fix undefined variable
Add changelog entry
dtmm: Fetch mod image from Nexus
lint/clippy Checking for common mistakes and opportunities for code improvement
build/msvc Build for the target platform: msvc
build/linux Build for the target platform: linux
When importing an archive file downloaded from Nexus, the file name does
include a version field. But, presumably for compatibility reasons,
Nexus replaces special characters with `-`, so that this field doesn't
match common schemes like `1.0.0`.
So instead we use the also included update timestamp to find the
corresponding file info from Nexus and use the version data from that.
Closes#131.
The initial implementation of DML ended up loading mods quite late,
which did give it the benefit of all `Manager`s being available.
This change therefore moves mod loading until after those are
initialized.
But contrary to old DML, we still create a separate game state to make
sure the game doesn't advance until mods are loaded. This avoids race
conditions like the one where LogMeIn needs to come early in the load
order.
Non-bundled mods come without a `dtmt.cfg`, and therefore without a
version number. But we need a version number at import to compare to
for the Nexus update check.
