CaveWhere can import survey notes from a PDF document

CaveWhere now supports importing PDFs for notes. Most survey trips produce multiple pages of notes and some scan their notes in as PDF. A PDF can keep all the raster notes together in one file. Currently, CaveWhere 0.09 only supports a variety of raster image formats like PNG, JPG and webp. Unlike 0.09, the development version of CaveWhere now supports multi-page PDF. CaveWhere will automatically convert these PDFs into raster files. By default, the PDF files are converted at 300ppi. This resolution is a good balance between disk space and quality. You can change default resolution in Settings->PDF. Since PDFs also support vectors, this new feature enables you to import your vector art, too.

If you’re interested in trying this out, you can build from source with pdf branch or become a Patreon member and get early access binaries!

Note: The 64bit version of CaveWhere does not suffer from these limitations. MacOS is always 64bit and has no memory limitations.

The limitation

When exporting maps to PDF, SVG, PNG, TIF, or JPG with a 32bit computer, addressable memory space is the limiting factor. For example, a 32bit windows application can allocate up to 2GB of RAM before running out of space.

It is recommended to image your computer’s memory as a bunch of labeled buckets. The label is an offset from the first bucket, so the first bucket would be bucket 0. The labels on the buckets go from 0 to 231 – 1 for a 32bit application (2,147,483,648 total labeled buckets). There are only 31 bits for labeling buckets, because the last bit stores the sign (i.e. negative or positive). If CaveWhere tries to use more than 231 buckets, it runs out of labels and generally, the operating system (aka Windows) will close the application. 64bit OS usually has much higher memory limits. Check out Windows memory limits to be sure.

Why is this important?

2.0GB is a ton of space.

With CaveWhere version 0.09 and above, CaveWhere uses significantly less memory than previous versions. CaveWhere uses a tiled base rendering process, allowing you to generate a map at any resolution. In previous versions before 0.09, CaveWhere did this all in RAM. In CaveWhere version 0.09 and above, the tiling is mapped to disk instead. This significantly reduces CaveWhere’s memory requirement. Even though CaveWhere does not use RAM for the map exporting, it still uses the same RAM address space, and this can be a problem for 32bit applications. For example, exporting 100in x 100in at 300pixel per inch map would require 3.35GB of temporary disk space. This easily goes past the 32bit application address limit. Different formats also have different limitations:

File Type Approximate 32bit limit 64bit limit
JPG ~225in2 (15in x 15in x 300 pixel/in) Free Space in RAM
PNG ~225in2 (15in x 15in x 300 pixel/in) Free Space in RAM
TIF ~225in2 (15in x 15in x 300 pixel/in) Free Space in RAM
SVG ~1089in2 (33in x 33in x 300 pixel/in) Free Space on Hard Drive
PDF ~2750 in2 (50in x 55in x 300 pixel/in) Free Space on Hard Drive

These are ballpark limitations. The limits will depend on the size of the cave and how much memory CaveWhere is already using. The results may vary. In the table above, CaveWhere was using about 250MB of RAM displaying an eight-mile cave.  

Out of the box, CaveWhere runs on a variety of platforms and computers. CaveWhere is implemented with OpenGL, a widely available and well supported graphics standard. Since OpenGL is a standard, it must be implemented by specific graphics vendors (such as NVidia, AMD, or Intel). Each vendors implementation of specific OpenGL extensions varies slightly. CaveWhere tries to detect the best possible rendering settings and use those settings to give you high performance rendering of your cave maps. This article tries to demystify the settings, shown below, to allow you to optimize CaveWhere for your computer.

CaveWhere Settings Page

Going to File->Settings (on Windows and Linux) or (CaveWhere -> Preferences) on MacOS, the Settings Page will appear. On this page, there is several rendering options, to play around with. Most of these options CaveWhere will apply them immediately, but a few require you to restart CaveWhere.

Job Settings

Job Settings allows you control the number of concurrent jobs that CaveWhere uses with a maximun number of threads. If you have a computer that’s less than 15 years old, it probably has multiple processing cores. You can see the number of available threads on your system described as “Usable threads:.” The system in the screenshot above has 12 available threads, meaning CaveWhere can do 12 jobs at the same time. By default, CaveWhere uses as many threads that exist on the system, giving you the best possible performance. If your computer overheats easily or you want to limit CaveWhere’s access to processing resources, you can reduce the access to the number of threads.

Texture Settings

Texture Compression Support

Compression support settings

By default, CaveWhere uses DXT1 texture compression, unless it is not supported. DXT1 texture compression reduces the amount of video memory and allows you to visualize six times more cave. Check out the Compress article for details.

GPU Texture Compression Support

GPU compression support

If your computer supports GPU Accelerated Compression, CaveWhere, by default, will use your GPU to compress textures. Turning this off will use a CPU implementation, which is slower, but higher quality than the GPU implementation. For more details, checkout the Compression article.

Anisotropy Support

Anisotropy support

If your graphics card supports Anisotropy, it is enabled by default. Anisotropy texture rendering improves the rendering quality by filtering textures and, overall, reduces rendering artifacts. It reduces rendering performance but should not be noticeable with most graphics cards. The illunstrations below show the difference between using Anisotropy and disabling it.

Example of Anisotropy enabled in Cavewhere

Anisotropy Enabled

Example of Anisotropy disabled in Cavewhere

Disabled Anisotropy

The differences are subtle and are more noticeable in CaveWhere compared to these screenshots. Anisotropy produces better rendering results by using mipmaps.

Mipmaps

Mipmaps

Mipmaps create a hierarchical pyramid of textures as shown in the image above. Using mipmaps greatly reduces rendering artifacts by enabling the graphics card to sample the image properly. Some graphics cards (specifically older Intel integrated chips) do not implement DXT1 Compression and Mipmap correctly. As a result, CaveWhere may get black images instead of a useful cave map. To fix the black rendering, disable Mipmaps or Texture Compression. In this case, if you want the best quality, disable Texture Compression and enable Mipmaps. See below for an illustration of the difference between enabled and disabled Mipmaps in CaveWhere.

Example of Mipmaps enabled in Cavewhere

Mipmaps Enabled

Example of Mipmaps disabled in Cavewhere

Mipmaps Disabled

If you cannot tell the difference in the images above, here’s a more concrete example below.

Mipmap Rendering

Magnification Filter

Magnification Filter controls how OpenGL samples texture pixels when you view a texture up close. CaveWhere has two Magnification Filter options: Nearest and Linear. Nearest will render texture such that you can see individual pixels where Linear will blend them together. See the image below to see the differences. Magnification Filter is more of a visual preference than anything else. 

Mag filter linear

Linear Magnification Filter. Check out how textures are blended together.

Mag filter nearest

Nearest Magnification Filter. CaveWhere renders the images as pixels.

Minification Filter

Minification Filter controls how Mipmap layers are blended. When you zoom out, CaveWhere must blend mipmap layers together. CaveWhere supports four different Minification Filters: Anisotropy, Linear Mipmap Linear, Nearest Mipmap Linear and Linear. By default, CaveWhere uses Anisotropy filtering if your graphics card supports it. Anisotropy filtering gives the rendering the best quality.

If your graphics card does not support Anisotropy filtering, then you have to choose amongst Linear Mipmap Linear, Nearest Mipmap Linear, and Linear filtering. Linear Mipmap Linear is tri-linear filtering, Nearest Mipmap Linear is bi-linear filtering, and Linear is well Linear filtering. If you want to know the details, here’s Wikipedia to the rescue. Below shows the differences amongst all four filtering modes.

Min filter linear

Linear Minification Filter

Min filter nearest linear

Nearest Linear Minification Filter

Min filter linear linear

Linear Linear Minification Filter

Although CaveWhere detects your rendering settings automatically, sometimes it fails to detect the best texture filtering technique. If you are having rendering issues, like black textures, or textures fading in and out when you zoom, I recommend disabling Mipmaps or changing the Minification Filter to Linear, or possibility disabling Texture Compression.

Engine Settings

Renderer Settings

To apply new Engine Settings, restart CaveWhere. The program supports three different renderers on Windows and two on MacOS and Linux. The first renderer, which is recommended, is OpenGL. If your system supports OpenGL renderer, it will give you the best support and best performance. Generally, when using Windows OpenGL renderer, graphics drivers have been installed.

If you have not installed your graphics drivers, CaveWhere will automatically fall back to OpenGL ES via DirectX. With OpenGL ES via DirectX, CaveWhere uses a graphics emulation layer called ANGLES to convert OpenGL ES to DirectX. This layer is only supported under Windows.

The final option is Software Renderer. The Software Renderer will always work but does not have DXT1 compression support and is by far the slowest renderer. The Software Renderer will use the CPU implementation of OpenGL and will never run on the graphics card. Software Rendering might be useful for running CaveWhere in a virtual machine. If CaveWhere crashes while checking OpenGL settings, by default it will fall back to Software Renderer.

Native Text Rendering

Use Native Text Rendering Settings

By default, CaveWhere uses Qt text rendering algorithm to render text in OpenGL and this option is disabled. Qt’s text rendering algorithm provides extremely fast and high quality text rendering. It’s recommended to keep this option disabled unless your computer is old and has a misbehaving graphics card. The image below shows an example of a misbehaving graphics card. If this happens, check the box to use Native Text Rendering. You can also try using OpenGL ES via DirectX to try to fix the artifacts.

Text rendering artifacts

Example of misbehaving graphics card rendering Qt texture rendering.

OpenGL Info

OpenGL Info in CaveWhere

If you are ever curious what OpenGL version and extension that CaveWhere is using, OpenGL Info can help. Also, if you ever report a rendering issue, you can select and copy this information. It might help with debugging CaveWhere. 

This release comes with 284 changes and improvements over the 0.08 release of January 31, 2016. CaveWhere has been in constant development since then with more than 111 changes yet to be released for 1.0. Stay tuned!

DOWNLOAD 0.09

Please consider supporting CaveWhere on Patreon if you are not already. Developing CaveWhere is not free. It costs hundreds of dollars per year on hosting fees, developer programs, and code signing. If you support CaveWhere through Patreon, you get access to early releases, priority support and the opportunity to vote on new features. You are charged when a new release comes your way, about every six months. Releases that are just bug fixes are free. If you want new CaveWhere releases, support this work on Patreon!

So, what is new?

Code Signing

Code signing protects you from hijacked CaveWhere binaries. It works by running a cryptographic checksum on the official released binaries. When you download CaveWhere from the Internet, this checksum is checked by Windows or MacOS. If this checksum passes, then it is guaranteed that the CaveWhere binary is official, since I, Philip Schuchardt, have personally signed it.

Windows Code Sign

Example of CaveWhere’s Code Sign on Windows

If the code is not signed, the operating system usually flags the binary and tries to prevent you from installing it. This is a good thing! On Windows, to get a code sign key, I had to verify my identity through a notary which I accomplished through a bank. On MacOS, my identity was verified through the Apple Developer program. The screenshots above and below show the code signing at work. Overall, pay close attention, and do not install CaveWhere versions that are not signed.

Rendering of Running Profile Maps

CaveWhere now allows you to create maps from Running Profiles. See the image below for the new drop down in the scrap options.  Select “Running Profile.”

Adding a Running Profile in CaveWhere

An Example of adding Running Profile in CaveWhere

CaveWhere will automatically carpet your running profile sketches to the survey line. For the running profile to work, you need at least two stations that are connected by a survey shot. For running profiles, I recommend using a single string of survey shots and avoiding junctions or forking shots.

This requirement is different than plan scraps. With plan scraps, CaveWhere assumes that survey shots are projected on a XY plane. This allows plan scraps to have loops and junctions in the same scrap. With running profiles, CaveWhere projects the scrap per shot, giving the wrapping effect along the cave survey. Check out the example below:

Visualization of Running Profile in CaveWhere

CSV Importer

CaveWhere can now import survey data directly from CSV files. It can import a single trip, or multiple trips, if the trip data is separated by a new line. It also supports custom column format and skipping columns. The import will warn you if you have done something wrong or the CSV file is not valid. Below is a screenshot of the CSV importer.

CaveWhere's CSV Importer

Ask to Save

CaveWhere now asks you if you would like to save before quitting or opening a new file. In 0.08, CaveWhere would just quit without asking and you might have lost some work. In 0.09, before quitting, CaveWhere temporarily saves the file, and checks it against the version on disk. If it is the same, it does not ask you, if it’s different, it pops up the dialog below.

Ask before Saving

High DPI Monitor Support

In 0.08, CaveWhere assumed a 96DPI display and did not honor screen scaling from Windows or MacOS. With 0.09, CaveWhere now scales text and rendering correctly.

New Threading Architecture and Progress Reporting

Many of CaveWhere’s algorithms have been ported to a new thread scheduling technology called asyncfuture. This allows CaveWhere to report more progress to the user.

Settings

CaveWhere now has a settings page. You can access it through File->Settings on Windows and Linux, and CaveWhere->Preferences on MacOS. Currently, most of the settings are rendering settings. Check out this article for more details.

CaveWhere Settings

Enhanced Texture Support

CaveWhere has always used a block compression algorithm called DXT1 compression for texture mapping. DXT1 compression provides a fixed 6:1 compression ratio. This ratio allows CaveWhere to display six times more cave as compared to uncompressed textures. In 0.09, CaveWhere now detects if your computer supports GPU compression. In 0.08, on Windows, CaveWhere would always use CPU compression. You can switch between GPU and CPU compression in the Settings Page. Check out this texture compression article for more details. In 0.09, you can also disable texture compression completely if you want the highest quality rendering. Disabling texture compression will reduce the amount of cave you can visualize.

File version checking

CaveWhere has always been forward and backward compatible. Backward compatibility means that you can open older CaveWhere in a newer version. On the other hand, forward compatibility is more complicated. This means that old versions of CaveWhere can open files created by new versions of CaveWhere. To support forward compatibility, CaveWhere ignores unknown fields found in the newer file. The only problem occurs when you go to save. In 0.08, CaveWhere would just delete these unknown fields potential and you might lose data. In 0.09, CaveWhere checks the file version, and alerts the user and prevents them from saving over a newer file.

Export Enhancements

Exporting the 3D rendering now supports SVG, PDF, TIF, and JPG. PNG was supported before 0.09. For exceptionally large image exports (100in x 100in), CaveWhere may require ~3.35 GB of memory. To reduce the RAM requirement, CaveWhere now uses out-of-core memory or disk-based memory. 32bit systems have significant memory limitations since they can easily reach their 2GB to 4GB addressable space limitation. Check out this article on 32bit limitations.

Disable Automatic Updates

CaveWhere automatically updates the rendering view while you enter data or carpet. There is no need to hit update or recompile. In 0.09, you can disable automatic updates by unchecking the automatic update button in the lower left corner. Disabling automatic updates may reduce the amount of processing because there is less reprocessing. If you’re using CaveWhere in the field, disabling automatic updates will save your laptop’s battery power. It’s also gives you more manual control over CaveWhere.

Automatic Updates in CaveWhere

Automatic Building with Travis

Travis now automatically builds CaveWhere on Linux and MacOS and runs the testcases. Travis is not useful unless you are a developer. It does help CaveWhere stay stable across platforms.You can check out the current builds

Minor Changes

Walls Importer
  • Walls importer warns about ignore splay shots
  • Walls importer accepts degrees as a unit
Compass Export
  • Fixed LRUD in compass export
  • Added support for 13- and 15-character format
Survex import
  • Fixed Imports LRUD incorrectly if leg is reversed
Map Exporter
  • Export add extension correctly and opens the exported file
Renderer
  • Better DXT1 support with OpenGL ES
  • Fixed black texture rendering issue
Carpeting
  • Added warning to user if the scale is invalid
  • Fixed profile calculation issue if station where added out of order
Other
  • Fixed many memory leaks
  • Fixed localization with “,” and “.” for data entry
  • Fixed crash when opening Read-only file

It takes a long time to load images in CaveWhere. Why is that?

The short answer is because of DXT1 compression. But first, let’s talk about compression in general. There’s several image compression formats out there. You might be familiar with some of them, such as: JPG, PNG and even newer ones such as webp.

These formats are great at compressing image data on a hard drive. However, when you view them, your computer must decompress the image into raw RGB data. Raw data takes tons of space in RAM! For example, look at the scanned survey data below.

Cave Survey Notes of First Trip in Phake Cave

An example of scanned cave notes compressed with JPG.

The graph below compares different image compression formats. The RAW RGB takes a significant amount of space!

Compare compression algorithms

DXT1 Compression to the Rescue

Creating cave maps in CaveWhere involves carpeting the survey notes by using scraps. Scraps are small clipped images from the original survey notes. If you’re interested in learning how to create a cave map in CaveWhere with the carpeting process, check out this tutorial.

CaveWhere wraps the scrap images with its carpeting process. For CaveWhere to render the cave, it must upload all scrap images to the graphics card memory. If you have a large cave, all the scraps images could be many, many megabytes or even gigabytes of data! If your computer runs out of graphics memory, the graphics driver must swap between RAM and graphics card memory, greatly reducing CaveWhere’s performance.

By default, CaveWhere uses DXT1 compression to reduce the scrap image memory requirement on the graphics card. DXT1 compression isn’t new. It was first patented in 1997 by S3 Inc. and is widely supported on most graphics cards. DXT1 is a lossy block compression that has a fixed compression ratio of 6:1.

From the previous example, the RAW RGB data would now only take 3.91MB instead 23.44MB of graphics memory. By using DXT1 format, CaveWhere can model six times more cave before running out of graphics card memory.

Downsides of DXT1 Compression

DXT1 compression is lossy. Lossy image compression means that the compression algorithm approximates the original image to save space. As a result, the image loses color accuracy and contains artifacts (see image below). Hopefully, these artifacts are so subtle that it’s hard to tell the difference between the compressed image and the original.

NVIDIA's DXT1 compression

Zoomed in to assent the DXT1 Compression artifacts. Nvidia’s DXT1 GPU compression.

Original RGB image

Compression Speed

Another downside is compression speed. DXT1 compression algorithm takes significant time. This is why CaveWhere is slow to load images, because it’s running the DXT1 compression algorithm. Luckily, CaveWhere supports two implementations of DXT1 compression. The first uses CPU with Squish to do the compression and the other is with the graphics card (GPU). Once compressed, the GPU can decompress the image at lighting speed.

In CaveWhere 0.08 and below, on Windows, CaveWhere only used squish (the CPU compression). Even though it’s multi-threaded, Squish takes 8.4 times slower than the GPU implementation (check out the graph below). On MacOS and Linux, CaveWhere uses GPU DXT1 compression by default.

For a reference, CaveWhere uses “Squish, image threaded, block threaded” and “GPU, image multithread” if your computer supports threaded OpenGL contexts (mainly Windows and Linux with good graphics cards). This is an unscientific benchmark of compressing ~7.5mi of cave. Results will vary depending on your graphics card or number of cores on your computer. I generated this  benchmark with 6 core / 12 thread Intel i7-8750H and Nvidia 1070 Max-Q graphics card.

Comparison Between DXT1 Compression Algorithms

Squish (CPU compression) has slightly better image quality than the graphics card but isn’t worth the wait. Below is a zoomed-in version to illustrate the differences between Squish and Nvidia’s DXT1 GPU compression. Can you tell the difference, see below?

Squish DXT1 Texture Compression

DXT1 Squish (CPU) Implementation has better image quality.

NVIDIA's DXT1 compression

Nvidia’s DXT1 GPU Implementation has faster compression times.

In CaveWhere 0.09, GPU compression is enabled by default on all operating systems, if supported by your graphics card. If you want to use the CPU compression instead, just uncheck GPU compression in Settings.

If your graphics card doesn’t support DXT1 compression or you don’t feel like waiting around for compression, you have the option to leave the image uncompressed (see settings). Leaving the images uncompressed will take six times as much GPU memory with no rendering artifacts. Since CaveWhere only compresses the images once, it’s recommended to keep compression on and reap its benefits!