Skip to content

Commit 84a2f7e

Browse files
loadkpiloadkpi
authored
update data from php lib v1.2.121 (#51)
- update data from php lib v1.2.121 - use _rubygems/release-gem_ - update paths in _bin/update_raw_files_
1 parent cc93353 commit 84a2f7e

File tree

8 files changed

+35
-10
lines changed

8 files changed

+35
-10
lines changed

‎.github/workflows/build.yml

+18-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name: build
22

3-
on: [push]
3+
on: [push, pull_request]
44

55
jobs:
66
lint:
@@ -9,7 +9,7 @@ jobs:
99
- uses: actions/checkout@v4
1010
- uses: ruby/setup-ruby@v1
1111
with:
12-
ruby-version: '3.0'
12+
ruby-version: ruby
1313
bundler-cache: true
1414
- run: bundle exec rubocop
1515
test:
@@ -30,3 +30,19 @@ jobs:
3030
- name: test
3131
run: |
3232
bundle exec parallel_rspec spec/ -n 4 --only-group ${{ matrix.test-group }} --group-by runtime --runtime-log spec/fixtures/parallel_runtime_rspec.log
33+
push:
34+
# Run on push to master branch
35+
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
36+
name: Push gem to RubyGems.org
37+
runs-on: ubuntu-latest
38+
permissions:
39+
id-token: write
40+
contents: write
41+
steps:
42+
- uses: actions/checkout@v4
43+
- name: Set up Ruby
44+
uses: ruby/setup-ruby@v1
45+
with:
46+
ruby-version: ruby
47+
bundler-cache: true
48+
- uses: rubygems/release-gem@v1

‎README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Comparing with other popular bot-detection gems:
1111
| | CrawlerDetect | Voight-Kampff | Browser |
1212
|--|--|--|--|
1313
| Number of bot-patterns | >1000 | ~280 | ~280 |
14-
| Number of checked HTTP-headers | 10 | 1 | 1 |
14+
| Number of checked HTTP-headers | 11 | 1 | 1 |
1515
| Number of updates of bot-list *(1st half of 2018)* | 14 | 1 | 7 |
1616

1717
In order to remain up-to-date, this gem does not accept any crawler data updates – any PRs to edit the crawler data should be offered to the original [JayBizzle/CrawlerDetect](https://github.com/JayBizzle/Crawler-Detect) project.

‎bin/update_raw_files

+2-2
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,5 @@ wget -O $DIR/../lib/crawler_detect/library/raw/Crawlers.json https://raw.githubu
66
wget -O $DIR/../lib/crawler_detect/library/raw/Exclusions.json https://raw.githubusercontent.com/JayBizzle/Crawler-Detect/master/raw/Exclusions.json
77
wget -O $DIR/../lib/crawler_detect/library/raw/Headers.json https://raw.githubusercontent.com/JayBizzle/Crawler-Detect/master/raw/Headers.json
88

9-
wget -O $DIR/../spec/fixtures/crawlers.txt https://raw.githubusercontent.com/JayBizzle/Crawler-Detect/master/tests/crawlers.txt
10-
wget -O $DIR/../spec/fixtures/devices.txt https://raw.githubusercontent.com/JayBizzle/Crawler-Detect/master/tests/devices.txt
9+
wget -O $DIR/../spec/fixtures/crawlers.txt https://raw.githubusercontent.com/JayBizzle/Crawler-Detect/master/tests/data/user_agent/crawlers.txt
10+
wget -O $DIR/../spec/fixtures/devices.txt https://raw.githubusercontent.com/JayBizzle/Crawler-Detect/master/tests/data/user_agent/devices.txt

‎lib/crawler_detect/library/raw/Crawlers.json

+1-1
Large diffs are not rendered by default.

‎lib/crawler_detect/library/raw/Exclusions.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
["Safari.[\\d\\.]*","Firefox.[\\d\\.]*"," Chrome.[\\d\\.]*","Chromium.[\\d\\.]*","MSIE.[\\d\\.]","Opera\\\/[\\d\\.]*","Mozilla.[\\d\\.]*","AppleWebKit.[\\d\\.]*","Trident.[\\d\\.]*","Windows NT.[\\d\\.]*","Android [\\d\\.]*","Macintosh.","Ubuntu","Linux","[ ]Intel","Mac OS X [\\d_]*","(like )?Gecko(.[\\d\\.]*)?","KHTML,","CriOS.[\\d\\.]*","CPU iPhone OS ([0-9_])* like Mac OS X","CPU OS ([0-9_])* like Mac OS X","iPod","compatible","x86_..","i686","x64","X11","rv:[\\d\\.]*","Version.[\\d\\.]*","WOW64","Win64","Dalvik.[\\d\\.]*"," \\.NET CLR [\\d\\.]*","Presto.[\\d\\.]*","Media Center PC","BlackBerry","Build","Opera Mini\\\/\\d{1,2}\\.\\d{1,2}\\.[\\d\\.]*\\\/\\d{1,2}\\.","Opera"," \\.NET[\\d\\.]*","cubot","; M bot","; CRONO","; B bot","; IDbot","; ID bot","; POWER BOT","OCTOPUS-CORE","htc_botdugls","super\\\/\\d+\\\/Android\\\/\\d+"]
1+
["Safari.[\\d\\.]*","Firefox.[\\d\\.]*"," Chrome.[\\d\\.]*","Chromium.[\\d\\.]*","MSIE.[\\d\\.]","Opera\\\/[\\d\\.]*","Mozilla.[\\d\\.]*","AppleWebKit.[\\d\\.]*","Trident.[\\d\\.]*","Windows NT.[\\d\\.]*","Android [\\d\\.]*","Macintosh.","Ubuntu","Linux","[ ]Intel","Mac OS X [\\d_]*","(like )?Gecko(.[\\d\\.]*)?","KHTML,","CriOS.[\\d\\.]*","CPU iPhone OS ([0-9_])* like Mac OS X","CPU OS ([0-9_])* like Mac OS X","iPod","compatible","x86_..","i686","x64","X11","rv:[\\d\\.]*","Version.[\\d\\.]*","WOW64","Win64","Dalvik.[\\d\\.]*"," \\.NET CLR [\\d\\.]*","Presto.[\\d\\.]*","Media Center PC","BlackBerry","Build","Opera Mini\\\/\\d{1,2}\\.\\d{1,2}\\.[\\d\\.]*\\\/\\d{1,2}\\.","Opera"," \\.NET[\\d\\.]*","cubot","; M bot","; CRONO","; B bot","; IDbot","; ID bot","; POWER BOT","OCTOPUS-CORE","htc_botdugls","super\\\/\\d+\\\/Android\\\/\\d+","\"Yandex\"","YandexModule2"]

‎lib/crawler_detect/library/raw/Headers.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
["HTTP_USER_AGENT","HTTP_X_OPERAMINI_PHONE_UA","HTTP_X_DEVICE_USER_AGENT","HTTP_X_ORIGINAL_USER_AGENT","HTTP_X_SKYFIRE_PHONE","HTTP_X_BOLT_PHONE_UA","HTTP_DEVICE_STOCK_UA","HTTP_X_UCBROWSER_DEVICE_UA","HTTP_FROM","HTTP_X_SCANNER"]
1+
["HTTP_USER_AGENT","HTTP_X_OPERAMINI_PHONE_UA","HTTP_X_DEVICE_USER_AGENT","HTTP_X_ORIGINAL_USER_AGENT","HTTP_X_SKYFIRE_PHONE","HTTP_X_BOLT_PHONE_UA","HTTP_DEVICE_STOCK_UA","HTTP_X_UCBROWSER_DEVICE_UA","HTTP_FROM","HTTP_X_SCANNER","HTTP_SEC_CH_UA"]

‎spec/fixtures/crawlers.txt

+8-1
Original file line numberDiff line numberDiff line change
@@ -1116,7 +1116,6 @@ ShowyouBot (http://showyou.com/crawler)
11161116
Yoleo Consumer v0.2
11171117
Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
11181118
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
1119-
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
11201119
Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
11211120
Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
11221121
CopperEgg/RevealUptime/DublinIE(aws)
@@ -2874,6 +2873,8 @@ COMODO DCV
28742873
CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
28752874
ZmEu
28762875
RSSOwl/2.2.1.201312301314 (Windows; U; en)
2876+
RuxitSynthetic/
2877+
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Safari/537.36 RuxitSynthetic/1.0 v119975163693 t1975984516666483988 ath87850d94 altpub cvcv=2 smf=0
28772878
WebClient/1.0
28782879
Microsoft-WebDAV-MiniRedir/10.0.10586
28792880
Microsoft-WebDAV-MiniRedir/10.0.14393
@@ -3669,3 +3670,9 @@ Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MicrosoftPreview/
36693670
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GeedoProductSearch; +http://www.geedo.com/product-search.html) Chrome/79.0.3945.88 Safari/537.36
36703671
Mozilla/5.0 (compatible; +probely.com/sos) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36 ProbelySPDR/0.2.0
36713672
facebookcatalog/1.0
3673+
Verity/1.1 (gumgum.com/verity; [email protected])
3674+
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/117.0.5938.132 Safari/537.36
3675+
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.5938.132 Mobile Safari/537.36 (compatible; GoogleOther)
3676+
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; GoogleOther)
3677+
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
3678+
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

‎spec/fixtures/devices.txt

+3-1
Original file line numberDiff line numberDiff line change
@@ -165632,4 +165632,6 @@ Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (K
165632165632
Mozilla/5.0 (iPhone; CPU iPhone OS 12_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148/mvmkbtaqqdhshwcbsygjnuqcqmpndbrt
165633165633
Mozilla/5.0 (Linux; Android 5.0; SM-N9100 Build/LRX21V) > AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 > Chrome/37.0.0.0 Mobile Safari/537.36 > MicroMessenger/6.0.2.56_r958800.520 NetType/WIFI
165634165634
Mozilla/5.0 (Linux; U; Android 5.0; en-US; SM-N900S;iPhone: Build/LRX21V)
165635-
Mozilla/5.0 (Linux; Android 9; OCTOPUS-CORE T8 Build/PPR1.181005.003) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 YaApp_Android/9.50/apad YaSearchBrowser/9.50
165635+
Mozilla/5.0 (Linux; Android 9; OCTOPUS-CORE T8 Build/PPR1.181005.003) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 YaApp_Android/9.50/apad YaSearchBrowser/9.50
165636+
Mozilla/5.0 (Linux; Android 9; YandexModule2-00001 Build/PI; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/113.0.5672.163 YaBrowser/24.1.2.86 (lite) TV Safari/537.36
165637+
Mozilla/5.0 (Linux; Android 9; YandexModule2-00001 Build/PI; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/113.0.5672.163 Mobile Safari/537.36

0 commit comments

Comments
 (0)