Access

Access Logs Plugin For Typecho

Record typecho system access logs and analytics.

Thanks

Thanks a lot for the ide support from jetbrains.

Intro

Current Version: CHANGELOG

Current Lauguage: English | Simplified Chinese

Preview

Latest Update Description

• Fix Chromium Style Error.
• Add IPv6 Support.
• Add MailMaster Support.
• General bug fixes and performance improvements.

Notice

• This branch need to change the Database to _access (REMOVE the _log suffix), Example: typecho.typecho_access.

• When the plugin update, please disable the plugin before updating.

• The plugin directory name must be 'Access'.

• Requires PHP 8.2 or newer (checked at activation).

• Needs the PHP cURL, intl, mbstring and GMP extensions, and a 64-bit PHP build. All of these are checked at activation and refused individually if missing.

• MySQL / MariaDB, SQLite and PostgreSQL are supported. The schema files live in sql/MySQL.sql, sql/SQLite.sql and sql/PostgreSQL.sql; the right one is picked automatically from the active Typecho database adapter.

• Access logs share Typecho's own database by default, but the plugin settings can point them at a separate MySQL / PostgreSQL server instead (host, port, user, password, database, table prefix). The target database must already exist — the plugin only creates the table. The connection is tested before the settings are saved, and article titles are resolved with a second query against Typecho's own database since a cross-database JOIN is not possible. Existing stats are migrated to the new database: up to 50k rows are moved inline when the settings are saved. Beyond that there are two options, both using keyset pagination with batched inserts and both resumable: a progress bar in the stats console, which drives the migration in ~3s chunks from the browser (no SSH needed, measured at roughly the same throughput as the CLI), and php usr/plugins/Access/tools/migrate.php for very large tables or scheduled jobs. An incomplete migration is always reported as incomplete — it is never silently treated as done. Sites upgrading from 3.0.x must disable and re-enable the plugin once so the migration route gets registered.

• When Redis is configured (i.e. "cache acceleration" is on), a write queue is enabled automatically: hits are pushed to a Redis list and flushed to the database in batches with a single multi-row INSERT. This removes the per-hit connection setup and cuts database connections from one-per-hit to one-per-batch, which is what actually saturates max_connections under a traffic spike. Measured against a separate PostgreSQL, one process per request: 4.67 ms/hit direct vs 0.12 ms/hit queued. Flushing is triggered by visits and deferred until after the response is sent (fastcgi_finish_request under PHP-FPM), the console flushes synchronously before reading, and tools/flush-queue.php can be put on cron as a backstop. The queue is never trimmed unless the write actually succeeded, and it is capped at 200k entries. With no Redis — or with the queue disabled in the settings — writes behave exactly as before.

• Large tables (hundreds of thousands of rows and up) get two extra covering indexes. The overview aggregates by time range: PV is COUNT(*), IP is COUNT(DISTINCT ip), UV is COUNT(DISTINCT ip, ua). With only a single-column time index the last two have to visit the heap for every matching row to read ip / ua, and on a 3M-row table that heap I/O dominates. Since 3.1.0 the schema ships with (time, ip) and (time, ip, ua); on PostgreSQL the plan becomes an Index Only Scan with Heap Fetches: 0 and buffer reads drop from 838 to 165. Tables created by 3.0.x are missing them; the plugin does not alter existing tables, so run the two CREATE INDEX statements from the schema file under sql/ by hand (substituting your own table prefix for typecho_). Use CONCURRENTLY on PostgreSQL to avoid blocking reads and writes; MySQL and SQLite block writes on that table while the index builds, so pick a quiet moment. Measured at roughly 6s per index on 3M rows.

• Whole-table aggregates (the three totals, the referrer Top N, the article pie chart) get their own caching rules, because the covering indexes only help queries with a time range and these have no WHERE at all — their cost grows linearly with the table and has no upper bound. They are served stale-while-revalidate: when the cached value expires it is still returned immediately and the recomputation is deferred to after the response is sent (fastcgi_finish_request), so nobody ever waits on one. On a completely cold cache nothing is computed inline at all — the endpoint answers done=false and the front end asks again a couple of seconds later, because a query with no upper bound can easily outlive nginx's fastcgi_read_timeout: the browser gets a 504 while PHP keeps grinding, and a few reloads are enough to fill the FPM pool. A mutex marker keeps repeated polling from starting more than one recomputation, and a failed run leaves that marker to expire on its own as a backoff. Flushing the write queue no longer invalidates any of these — a handful of new hits is a rounding error on an all-time total. The per-day and per-month keys are still invalidated, which is what that logic was added for in the first place.

• The console renders on the client, so the JSON endpoints return raw data and escaping belongs to the renderer. Never HTML-escape a value on its way into a JSON response — doing it twice makes the page display literals like ' instead of quotes. In the log table the referrer is only rendered as a link when it starts with http:// or https://, and as plain text otherwise: the Referer header is entirely attacker-controlled, and escaping does not stop a javascript: URL from being a link that runs on click.

• The overview page loads in sections — today, yesterday, totals, referrers, the article pie chart and the current month are each their own request, filling the page as they arrive. This breaks a vicious cycle: one big request times out, so nothing finishes, so nothing is cached, so the next load is a first load all over again. Split up, every request completes (the slowest section measured 1.5s on 3M rows), the cache gets built, and a warm overview is around 0.1s. Past days are cached long-term since their numbers can no longer change, and the month chart advances day by day, resuming from the cache if one request cannot finish it. That resumption needs Redis; without it the sections still work, they are just recomputed each time.

• Since v3.1.2, plugin settings can live in a file and ship with the code. On activation, if config/current.yaml exists the plugin configures itself from it; on deactivation the current settings are written back to that same file, so "deactivate → move host or rebuild the container → put the file back → activate" is a closed loop with no re-typing in the admin UI. Keys present in the file win, keys absent fall back to defaults — loading it replaces the configuration wholesale with what the file describes. The document is a top-level access: map of key: value lines; see config/README.md for every key, its default and a full example. Booleans accept 1/0 or true/false, and quoted values are always literal (dbName: "off" is the string off). Only a small YAML subset is supported (scalars, quotes, comments) and no extension is required, though PHP's yaml extension is used for parsing when present. The file holds the database password, Redis password and IPinfo token: it is written with mode 0600, and config/ ships with an .htaccess and an empty index.html to block direct download — neither has any effect under Nginx, so deny that directory in the site config yourself; the plugin's .gitignore already lists config/current.yaml so it is not committed by default. A parse failure (bad syntax, unreadable file) never blocks activation; it is reported in the success notice and the existing configuration is kept. A write failure (read-only directory) never blocks deactivation. The table is created after the file's settings are applied, so a separate database named in the file takes effect immediately — and if that database cannot be reached, activation fails as usual without overwriting the configuration already stored in the database.

• Since v3.1.3, when Redis is enabled in the configuration, both activation and saving the settings probe it once. Redis is only an accelerator, so an unreachable Redis never blocks activation — the cache and the write queue degrade automatically, statistics stay complete, and the notice simply says Redis is configured but currently unavailable; once it comes back the plugin reconnects on its own. The connect timeout is 0.5s and is backed by a circuit breaker: after a failed connection the plugin degrades directly for 30 seconds instead of retrying. That breaker is not optional polish — Core is constructed in Widget\Archive::beforeRender, and an address that silently drops packets (firewall, DNS blackhole, unreachable container network) burns the full timeout, so without it every front-end request pays that cost. Measured against a blackhole address: 502 ms without the breaker, 0 ms with it. The breaker state has to survive between requests — PHP starts each one from a clean slate — so it is kept in APCu when that extension is present (shared memory, native TTL) and otherwise in a marker file under the system temp directory, using its mtime as the timestamp. APCu is an optional dependency; without it nothing is lost but that one layer of speed. The marker lives in a 0700 directory named after a fingerprint of the plugin path, so several sites on one host do not collide. A refused connection (nothing listening on the port) returns in milliseconds and was never the problem; the breaker exists for the "no response" case.

• Since v3.1.5 the write queue has been hardened around one goal: no hit disappears silently. Messages now travel through three Redis lists — the queue, processing (being written), and a dead-letter list. The consumer claims a batch with a small Lua script that moves it out of the queue atomically, instead of reading it, writing it, and then trimming by position: the producer's own trim (which fires once the queue passes 200k entries) used to be able to land between the read and the trim, at which point the consumer trimmed away messages that had just arrived and were never written. Crashing after a claim no longer loses anything either — that batch sits in processing and the next flush picks it up first. The flush lock now uses a random token, is released through a Lua compare-and-delete, renews itself every 10s while writing, and stops immediately if it finds the lock has changed hands; it is also acquired in shutdown, when the flush actually starts, rather than before the page renders. Previously the lock value was the process PID and release was a plain DEL, so a timed-out consumer would delete the lock a new consumer was holding, putting several consumers on the same queue. Rows the database refuses now go to a dead-letter list (plugin:access:{fingerprint}:queue:dead, capped at 10k) with their original payload rather than being dropped along with the batch — one success out of a thousand used to acknowledge and delete the other 999. When a whole batch fails the plugin probes the connection first, so "database is down" (keep and retry) is told apart from "this batch is all bad data" (move to dead letters); a single unwritable row at the tail of the queue no longer wedges it forever. The per-flush cap counts messages taken, not rows successfully written, and there is a 20s wall-clock cap on top. After a successful flush the statistics caches are invalidated for exactly the affected dates — previously only a timestamp was updated, so a queue flushed after midnight left the previous day's numbers wrong until the long cache expired on its own, up to 40 days later. Redis connections are all created in one place and now carry a read timeout (3s in the front end, 5s on the CLI); with only a connect timeout, a peer that completes the TCP handshake and then goes silent stalled until PHP's default socket timeout — measured at 60.06s against a peer that accepts the connection and then never answers, versus returning on the timeout once one is set. Log fields are truncated to the column widths in the schema and cid / mid are clamped to int unsigned, and the front-end GIF endpoint is rate limited to 60 requests per IP per minute: that endpoint is reachable anonymously, and one oversized record used to fail the batch INSERT and degrade it into a thousand single-row INSERTs. Deactivating the plugin no longer deletes data that never reached the database — it drains the queue in a loop first, and if anything is left while "drop data" is off, the queue and dead letters are kept as they are and resume on the next activation. Saving settings that would move the queue's home (Redis address or password, turning Redis off, turning the write queue off, changing the stats database or table prefix) drains the backlog under the old configuration before saving. tools/flush-queue.php exits 1 when the database is unavailable, the flush is interrupted, the lock changes hands, or messages went to dead letters (--lenient waives the last), sends errors to STDERR, gains a --deadline option, and no longer reports a broken Redis as "queue is empty" and exits successfully.

• Since v3.2.0 the table carries a schema version, and upgrades alter it automatically. The version is stored in Typecho's own options table, recorded per stats database, so switching between several stats databases keeps each one's structure version straight. Activating the plugin — or simply saving the settings once — compares it against the code and runs the pending upgrade steps (ALTER TABLE to add a column, CREATE INDEX to add an index); a run that fails partway stops at the last step that succeeded and resumes from there next time. The number only follows the plugin version when the schema actually changes, and stays put otherwise, so the vast majority of releases do not pay for an upgrade check they do not need; it currently reads 3.2.2. Upgrading from 3.1.x needs nothing more than the usual deactivate-then-activate. If you only swapped the plugin files and have not re-activated yet, writes automatically drop the new column and carry on (without idempotency for that window) rather than failing the whole batch — the upgrade itself must never break statistics.

• With the write queue on, Redis is no longer a cache you can throw away — it holds hits that have not reached the database yet, so configure it as a datastore. Turn persistence on (appendonly yes, appendfsync everysec or always); with RDB snapshots alone, everything written between snapshots is gone after a restart, which means a stretch of access logs simply vanishes. Do not give this Redis an eviction policy that can drop queue keys — allkeys-lru / allkeys-random will evict the queue itself under memory pressure and leave no trace; use a volatile-* policy (the queue, processing and the dead-letter list carry no TTL, only the statistics caches do) or noeviction. The loss window is up to one second of writes with appendfsync everysec if the process is killed, plus whatever has not replicated yet if a replica is promoted; lower queueFlushSize / queueFlushInterval to shorten the time data spends in Redis. Only standalone Redis is supported — not Redis Cluster: the claim script touches ...:queue and ...:queue:processing in one call and those keys carry no hash tag, so Cluster answers CROSSSLOT. A single writable address behind Sentinel works, subject to the failover window above.

• Redis keys carry a site fingerprint (plugin:access:{12 hex}:..., derived from the plugin directory path), so several Typecho installs sharing one Redis database no longer share a queue, a flush lock or a cache. Since v3.2.3 the prefix is plugin:access: rather than typecho_access:, matching the plugin:{name}: convention used by the other plugins on this site. The prefix change is a hard cut, not an adoption: drain the queue before upgrading — run tools/flush-queue.php, or deactivate the plugin once from the admin panel — or the messages still sitting in typecho_access:{fingerprint}:queue will be stranded there with no consumer. Older, pre-fingerprint keys (typecho_access:queue and friends) are still adopted automatically: save the plugin settings once, or let tools/flush-queue.php run. Sites that share one plugin directory (symlinks, a shared code checkout) still produce the same fingerprint and still collide — give those a separate Redis instance. Uninstall cleanup scans both plugin:access:* and typecho_access:*, but only deletes keys under this site's fingerprint plus the pre-fingerprint legacy keys; other sites on the same Redis are left alone.

• Settings that change where the queue belongs now require a clean queue. The Redis host/port/password, the write-queue switch, and the stats database type/host/name/prefix all decide who owns the messages already sitting in the queue: change Redis and they become unreachable, change only the database and they get written to the wrong one. The plugin drains the backlog under the old configuration first and refuses to save if anything is left, explaining which of the two cases applies. To go ahead and abandon those messages, set "queue ownership change" to "force once" and save — it applies to that one save and resets itself afterwards.

• PostgreSQL note: the schema pins n_distinct = -0.1 on the ip column, and the upgrade step adds it to existing tables. ANALYZE extrapolates n_distinct from a 30k-row sample and is notoriously unreliable for high-cardinality columns — measured on a 3.14M-row table with 450k distinct IPs (14.3%), it estimated 22,878 (0.73%), a 20× underestimate. The consequence is not a slightly-off number but the wrong plan: the planner sized a hash table for 23k groups when 450k were needed, and once past work_mem the HashAggregate spilled to disk and repartitioned. The overview's all-time distinct-IP count took 643 seconds and the console reliably returned 504; with the estimate corrected the planner switches to deduplicating along the (ip, ua) index order with no hash table at all, and it takes 1 second. Direction matters more than precision here — underestimating spills and costs hundreds of times, overestimating just reads a bit more index and is bounded — so the value errs high. To tune it for your own data, measure once with SELECT count(DISTINCT ip)::float / count(*) FROM <prefix>access;. The setting is lost whenever the table is rebuilt and gives no sign of it (nothing breaks, one query just gets hundreds of times slower), so it is verified on every activation alongside the event_id unique index.

• The ip column stores the address as its decimal integer representation (ip2long for IPv4, a gmp conversion for IPv6) — not as text. The width is therefore bounded by the digit count of 2^128-1, which is 39. Earlier versions used 38, and dropping the last digit of a decimal string divides it by ten, so fc00::/7, fe80::/10 and ff00::/8 — the ranges whose decimal form is exactly 39 digits — were silently stored as a different, plausible-looking address. 2000::/3 global unicast is 38 digits and was never affected, which is why this went unnoticed for so long; fe80:: shows up as REMOTE_ADDR often enough behind a reverse proxy or on a LAN. The v3.2.1 schema step widens the column to char(39) on existing MySQL tables; sql/PostgreSQL.sql has always been varchar(39) and SQLite does not enforce character lengths, so neither needs touching. Pick a quiet moment on large MySQL tables: changing a CHAR length can only use ALGORITHM=COPY, which rebuilds the table and every index and blocks writes. The step probes the current width first and does nothing when the column is already wide enough, so it never rebuilds a table twice. Re-activate the plugin (or save the settings once) soon after upgrading the files: in the window where the new code is running against the old column, those addresses are rejected by MySQL strict mode and go to the dead-letter queue instead of the stats table — nothing is lost, and flush-queue.php exits 1 to say so.

• Since v3.2.4 an unrecognised User-Agent counts as a robot. Detection runs six checks, most-certain first, and any hit means robot: (1) known tools (curl, wget, masscan, python-requests, okhttp, Java, …); (2) the curated crawler list; (3) a name containing bot / spider / client / User; (4) a URL or an email address inside the UA — putting contact details in the UA (+http://…) has been crawler convention for two decades and real browsers have no reason to do it, so the URL's host or the mail domain becomes the robot name; (5) an empty or placeholder UA; (6) the fallback: neither the browser nor the OS could be identified. Before this only (2) and (3) existed, and anything they missed was recorded as a human — which is how Hello from Palo Alto Networks … https://…, and more importantly curl, wget and empty UAs, ended up counted as real visitors. Check (6) is the one that can misfire, and how often depends on how complete the browser/OS rules are; the trade is deliberate — a niche browser filed as a robot is visible the moment you look at the UA, whereas scanners sitting in your human traffic quietly inflate every number. The crawler list now matches on word boundaries: it used to lowercase and strip whitespace and then substring-match, so Custo matched the word customers inside a long UA and Ask matched anything containing "ask". Presto (Opera 12's layout engine) and TencentTraveler are real browsers and have been removed from the list — please do not add them back. Existing rows still carry the old classification; tools/reclassify-robots.php recomputes them (--dry-run first to see what would change, --limit plus --from to run it in chunks, and repeated runs are idempotent).

• Note that the overview statistics do not exclude robots. Unique IPs, UVs, PVs, the referer Top N and the post pie chart are all queried without filtering on robot; that column currently only drives the human/robot filter and the display name in the log list. So fixing robot detection and backfilling existing rows changes none of the overview numbers — they always counted robots. For the same reason the backfill needs no Redis cache invalidation.

• Since v3.2.0 writes are idempotent, so a queue replay no longer produces duplicate rows. The table gains an event_id column with a unique index, and each hit is assigned a 32-character identifier when it is enqueued, reused on every retry — generating it at write time instead would make every retry a "new" record and leave the unique index doing nothing. The first 16 hex characters contain a millisecond timestamp plus 16 random bits and the rest is random, so identifiers cluster by millisecond and index writes stay near the right edge of the B+ tree rather than landing on a random page across the whole table. Identifiers generated within the same millisecond remain randomly ordered. The queue delivers at least once — a process killed after the database write but before the batch is acknowledged out of processing replays that batch — and the unique index turns that into exactly-once in effect. Inserts now skip on a unique conflict: ON DUPLICATE KEY UPDATE on MySQL, ON CONFLICT DO NOTHING on PostgreSQL, INSERT OR IGNORE on SQLite. MySQL deliberately does not use INSERT IGNORE: it also swallows data errors such as an over-long value, and the dead-letter queue depends on the database rejecting those explicitly. Existing rows keep a NULL event_id; all three databases allow several NULLs in a unique index, so old data is untouched — it simply has no idempotency of its own.

• v3.1.5 also fixes an inverted condition in front-end tracking mode: with writeType=0 (front end) the GIF endpoint was not recording anything, while with writeType=1 (back end) that endpoint could still be called anonymously to write extra rows. Sites using front-end tracking should upgrade.

Features

• Show PV/UV and more.
• Ignore the administrator login log.
• Support the referer and domain show and sort.
• Add remove all logs when the plugin is disabled feature.
• Support Frontend or Backend write log.
• Log filter supports filtering by ip, article title, and route
• Works on MySQL / MariaDB, SQLite and PostgreSQL

Author

And origin authors

@kokororin

@一名宅

@Zhizheng Zhang