Merge branch 'master' into fix-testing

Author: oldmoe

BENCHMARKS.md

@@ -125,16 +125,16 @@ On average, Litecable is quite faster than the Redis based version and offers be
 
 > ![litesearch](https://github.com/oldmoe/litestack/blob/master/assets/litesearch_logo_teal.png?raw=true)
 
-Litesearch was benchmarked against Mielsearch, both using their respective ActiveRecord integrations. Mielsearch was running on the same machine as the benchmark script and was using the default configuration options. The dataset used for testing was the infamous Enron email corpus. Redisearch was not benchmarked due to the clients being not Rails 7.1 compatible (yet), will probably bench Redisearch when they are.
+Litesearch was benchmarked against Meilisearch, both using their respective ActiveRecord integrations. Meilisearch was running on the same machine as the benchmark script and was using the default configuration options. The dataset used for testing was the infamous Enron email corpus. Redisearch was not benchmarked due to the clients being not Rails 7.1 compatible (yet), will probably bench Redisearch when they are.
 
 ### Building the index
 
-||MieliSearch|Litesearch|
+||Meilisearch|Litesearch|
 |-:|-:|-:|
 |Time to insert 10K docs|265.42 seconds|29.06 seconds|
 |Inserted docs/second|38|344|
 |Search latency (3 terms)|7.51 ms| 0.051ms|
 |Searches/second|133|19608|
 |Index rebuild|0.822|0.626|
 
-We only limited the test to 10K documents becuause MieliSearch was taking a long time to index, so we decided to stop at a reasonable sample size. The search numbers for litesearch were double checked, event against a 100K document set and they remained virtually the same. It is clear that litesearch is much faster than MieliSearch in both indexing and searching, this could be partially attributed to litesearch being a simpler text search engine, but still, the difference is huge! For rebuilding the index though, Litesearch is not that much faster than Mielisearch.
+We only limited the test to 10K documents becuause Meilisearch was taking a long time to index, so we decided to stop at a reasonable sample size. The search numbers for litesearch were double checked, event against a 100K document set and they remained virtually the same. It is clear that litesearch is much faster than Meilisearch in both indexing and searching, this could be partially attributed to litesearch being a simpler text search engine, but still, the difference is huge! For rebuilding the index though, Litesearch is not that much faster than Meilisearch.

CAVEATS.md

@@ -0,0 +1,20 @@
+# CAVEATS
+
+This is a document that lists some caveats that you need to consider when dealing with Litestack and SQLite applications
+
+## Single host (server) only 
+
+By design, SQLite can be accessed only (in a performant and safe way) from a single host machine. That means that you cannot have things like auto-scaling of app servers since there is but a single one of them. This app server can be a huge one, with many processes, but only one app server at a time. 
+
+## Containerization becomes a bit tricky
+
+Container technology was originally designed for stateless applications. Based on this, one of the core aspects of containers is that they are immutable, deploying an app to container means destroying one container and creating another. With statefull applications, like database servers and similarly SQLite based applications, this will potentially mean a down time will be perceived, which in some approaches is mitigate but not perfectly (yet). Expect some quirks when dealing with SQLite and containers.
+
+## Single writer / multi reader 
+
+In its default configuration, Litestack allows one writer at a time to write to the database file, while simultaneously allowing any number of readers to access the database. Generally SQLite is pretty fast in writes, but you should be aware that a long running write operation will prevent any other writers from proceeding, for example creating an index on a very large table will stop other writers until the index creation is finished. There is no concurrent index creation facility in SQLite, yet.
+
+## Litestack does not release the GVL
+
+This also applies to the Ruby SQLite3 gem, when it is performing a query, it will not allow any other threads in the Ruby process to resume until it returns. Care should be taken when writing low latency multi-threaded applications with Litestack, it is generally recommended to use fibers over threads in that case, since everything is serialized anyway it makes sense to avoid the ovreheads of mult-threaded environments 
+ 

FILESYSTEMS.md

@@ -0,0 +1,55 @@
+# Filesystems & Litestack
+
+In the containerized world we live in, many layers of the hardware/software stacks are far abstracted that we no longer know they exist. For the filesystem enthusiasts out there this is a quick overview of how Litestack (and hence SQLite) can benefit from different filesytems 
+
+## XFS
+
+A very stable and trusted filesystem with excellent performance charactersitcs 
+
+- Fast reads / writes
+
+## EXT4
+
+Another stable and performant filesystem
+
+- Fast reads / writes
+
+## F2FS 
+
+Specially built for solid state storage, has an atomic write mode that is supported by SQLite
+
+- Fast reads
+- Reasonably fast durable writes in synchronous mode
+- Compression (not very useful)
+
+## ZFS
+
+Copy-on-write filesystem with a nifty set of features, very fast snapshotting but only for the FS as a whole, can send incremental snapshots to another host for backup. Naturally a lot more pages are written on write operations thus it is a bit slower in writes.
+
+- Fast reads
+- Slower writes
+- FS Snapshotting with send/recv
+- fast device cache
+- Compression
+
+## Btrfs 
+
+Another CoW filesystem that delivers snapshotting and incremental send/recv at a much granular level.
+
+- Fast reads
+- Slower writes
+- Fast copies (free backups)
+- Sub volume Snapshotting with send/recv
+- Compression
+
+## Bcachefs
+
+A new CoW filesystem built on the foundations of the bcache module. Improving rapidly but still lacks a few of the common CoW fs features
+
+- Fast reads
+- Slower writes
+- fast device cache
+
+
+
+ 

README.md

@@ -59,7 +59,7 @@ litedb is a wrapper around SQLite3, offering a better default configuration that
 
 #### Direct litedb usage
 
-litedb can be used exactly as the SQLite3 gem, since litedb iherits from SQLite3
+litedb can be used exactly as the SQLite3 gem, since litedb inherits from SQLite3
 
 ```ruby
 require 'litestack'
@@ -209,7 +209,7 @@ idx.search('kamal')
 idx.search('subject: awa') 
 ```
 
-Litesearch integrates tightly with ActiveRecord ans Sequel, here are integration examples
+Litesearch integrates tightly with ActiveRecord and Sequel, here are integration examples
 
 #### ActiveRecord
 

bin/liteboard

@@ -4,6 +4,7 @@
 require 'optparse'
 require 'erb'
 require 'yaml'
+require 'rackup'
 require_relative '../lib/litestack/liteboard/liteboard'
 DEFAULTS = {
   config_path: Litemetric::DEFAULT_OPTIONS[:config_path],
@@ -17,7 +18,7 @@ options = {
   Port: 9292,
   Host: 'localhost',
   environment: 'production',
-  pid: 'tmp/pids/liteboard.pid',
+  pid: './liteboard.pid',
   quiet: false
 }
 
@@ -58,7 +59,7 @@ if options[:config_path]
     exit
   end 
 else # no config path! use the default
-  config = Yaml.load_file(DEFAULTS[:config_path]) rescue nil 
+  config = YAML.load(ERB.new(File.read(DEFAULTS[:config_path])).result) rescue nil 
 end
 
 if config
@@ -81,4 +82,4 @@ options[:app] = Liteboard.app
 require_relative '../lib/litestack'
 puts "Starting Liteboard version #{Litestack::VERSION}"
 
-Rack::Server.start(options)
+Rackup::Server.start(options)

lib/active_support/cache/litecache.rb

@@ -1,4 +1,5 @@
 require "delegate"
+require "active_support"
 require "active_support/core_ext/enumerable"
 require "active_support/core_ext/array/extract_options"
 require "active_support/core_ext/numeric/time"
@@ -73,6 +74,14 @@ def stats
 
       private
 
+      def serialize_entrys(entry, **options)
+        Marshal.dump(entry)
+      end
+
+      def deserialize_entrys(entry)
+        Marshal.load(entry.to_s)
+      end
+
       # Read an entry from the cache.
       def read_entry(key, **options)
         deserialize_entry(@cache.get(key))
@@ -94,15 +103,15 @@ def write_entry(key, entry, **options)
       def write_multi_entries(entries, **options)
         return if entries.empty?
         entries.each_pair {|k,v| entries[k] = serialize_entry(v, **options)}
-        expires_in = options[:expires_in].to_i
+        expires_in = options[:expires_in].to_i if options[:expires_in]
         if options[:race_condition_ttl] && expires_in > 0 && !options[:raw]
           expires_in += 5.minutes
         end
         @cache.set_multi(entries, expires_in)        
       end
 
       def write_serialized_entry(key, payload, **options)
-        expires_in = options[:expires_in].to_i
+        expires_in = options[:expires_in].to_i if options[:expires_in]
         if options[:race_condition_ttl] && expires_in > 0 && !options[:raw]
           expires_in += 5.minutes
         end

lib/litestack/liteboard/liteboard.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require "hanami/router"
+require "rack"
 require "tilt"
 require "erubi"
 
@@ -11,37 +11,32 @@ class Liteboard
   @@resolutions = {"minute" => [300, 12], "hour" => [3600, 24], "day" => [3600 * 24, 7], "week" => [3600 * 24 * 7, 53], "year" => [3600 * 24 * 365, 100]}
   @@res_mapping = {"hour" => "minute", "day" => "hour", "week" => "day", "year" => "week"}
   @@templates = {}
-  @@app = Hanami::Router.new do
-    get "/", to: ->(env) do
+  @@app = proc do |env|
+    case path = env["PATH_INFO"]
+    when "/"
       Liteboard.new(env).call(:index)
-    end
-
-    get "/topics/Litejob", to: ->(env) do
+    when "/topics/Litejob"
       Liteboard.new(env).call(:litejob)
-    end
-
-    get "/topics/Litecache", to: ->(env) do
+    when "/topics/Litecache"
       Liteboard.new(env).call(:litecache)
-    end
-
-    get "/topics/Litedb", to: ->(env) do
+    when "/topics/Litedb"
       Liteboard.new(env).call(:litedb)
-    end
-
-    get "/topics/Litecable", to: ->(env) do
+    when "/topics/Litecable"
       Liteboard.new(env).call(:litecable)
     end
+
   end
 
   def initialize(env)
     @env = env
-    @params = @env["router.params"]
+    @req = Rack::Request.new(@env)
+    @params = @req.params 
     @running = true
     @lm = Litemetric.instance
   end
 
   def params(key)
-    URI.decode_uri_component(@params[key].to_s)
+    URI.decode_uri_component(@params[key.to_s].to_s)
   end
 
   def call(method)
@@ -227,9 +222,9 @@ def litejob
     @topic = "Litejob"
     @events = @lm.events_summaries(@topic, @resolution, @order, @dir, @search, @step * @count)
     @events.each do |event|
-      data_points = @lm.event_data_points(@step, @count, @resolution, @topic, event["name"])
+      data_points = @lm.event_data_points(@step, @count, @resolution, @topic, event[:name])
       event["counts"] = data_points.collect { |r| [r["rtime"], r["rcount"] || 0] }
-      event["values"] = data_points.collect { |r| [r["rtime"], r["rtotal"] || 0] }
+      event["values"] = data_points.collect { |r| [r["rtime"], (r["rtotal"] || 0.0)] }
     end
     @snapshot = read_snapshot(@topic)
     @size = begin
@@ -391,6 +386,7 @@ def round(float)
   end
 
   def format(float)
+    float = float.round(3)
     string = float.to_s
     whole, decimal = string.split(".")
     whole = whole.chars.reverse.each_slice(3).map(&:join).join(",").reverse

lib/litestack/litecache.rb

@@ -162,13 +162,13 @@ def delete(key)
   end
 
   # increment an integer value by amount, optionally add an expiry value (in seconds)
-  def increment(key, amount, expires_in = nil)
+  def increment(key, amount=1, expires_in = nil)
     expires_in ||= @expires_in 
     @conn.acquire { |cache| cache.stmts[:incrementer].execute!(key.to_s, amount, expires_in) }
   end
 
   # decrement an integer value by amount, optionally add an expiry value (in seconds)
-  def decrement(key, amount, expires_in = nil)
+  def decrement(key, amount=1, expires_in = nil)
     increment(key, -amount, expires_in)
   end
 

lib/litestack/litejobqueue.rb

@@ -141,15 +141,16 @@ def stop
 
   def exit_callback
     @running = false # stop all workers
-    return unless @jobs_in_flight > 0
-    puts "--- Litejob detected an exit, cleaning up"
-    index = 0
-    while @jobs_in_flight > 0 && index < 30 # 3 seconds grace period for jobs to finish
-      puts "--- Waiting for #{@jobs_in_flight} jobs to finish"
-      sleep 0.1
-      index += 1
+    if @jobs_in_flight > 0
+      puts "--- Litejob detected an exit, cleaning up"
+      index = 0
+      while @jobs_in_flight > 0 && index < 30 # 3 seconds grace period for jobs to finish
+        puts "--- Waiting for #{@jobs_in_flight} jobs to finish"
+        sleep 0.1
+        index += 1
+      end
+      puts " --- Exiting with #{@jobs_in_flight} jobs in flight"
     end
-    puts " --- Exiting with #{@jobs_in_flight} jobs in flight"
   end
 
   def setup
@@ -179,6 +180,7 @@ def schedule(spawn = false, &block)
 
   # create a worker according to environment
   def create_worker
+    return if defined?(Rails) && !defined?(Rails::Server)
     Litescheduler.spawn do
       worker_sleep_index = 0
       while @running

lib/litestack/litemetric.rb

@@ -148,10 +148,11 @@ def run_stmt_hash(stmt, *args)
   end
 
   def exit_callback
-    return unless @collector.count > 0
-    warn "--- Litemetric detected an exit, flushing metrics"
     @running = false
-    @collector.flush
+    if @collector.count > 0
+      warn "--- Litemetric detected an exit, flushing metrics"
+      @collector.flush
+    end
   end
 
   def setup
@@ -203,8 +204,8 @@ def collect_metrics
     def create_snapshotter
       Litescheduler.spawn do
         while @running
-          sleep @litemetric.options[:snapshot_interval]
           capture_snapshot
+          sleep @litemetric.options[:snapshot_interval]
         end
       end
     end

lib/litestack/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Litestack
-  VERSION = "0.4.3"
+  VERSION = "0.4.4"
 end

litestack.gemspec

@@ -33,7 +33,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency "sqlite3"
   spec.add_dependency "oj"
   spec.add_dependency "rack"
-  spec.add_dependency "hanami-router"
+  spec.add_dependency "rackup"
   spec.add_dependency "tilt"
   spec.add_dependency "erubi"