Measuring the Web
Measure of an Organization
The 21st century has accelerated a 20th century trend – the rate of change across the World from technology thru the environment to social norms and governance – all are increasingly challenging to manage The Pandemic has heightened the issues as organizations find they must revisit their mission and strategy. Suddenly, organizations have had to change how they reach customers, deliver their services, support their own staff and keep their shareholders and suppliers in the know about their latest operational directions..
So it should not be a surprise that an Oranization’s Web systems now account for 1 in 7 direct sales and are instrumental in overall marketing, delivery and support operations. Thus, keeping tabs on your Website is actively supported by a wide array of tools tracking your website’s performance.
Website Measurement ToolsThere are many measurement tools for providing broad website performance data. First are tools for overall KPI-Key Performance Indicators like security, mobile responsiveness, speed of operations, SEO status, and detailed usage statistics. Then there are numerous specialized tools that measure all aspects website performance such as web security, website speed, SEO success and data integrity. As you might expect, there is overlap of features even among different measurement tool categories. Overall website measurement tools provide a macro benchmark of how your own or competitor websites are doing. As you can see from the table below the number of tests for each measurement category varies widely among the tools; but rest assured each of the tools has detailed analysis and tips for users to track and improve their websites.
Website Security MeasuresGiven the average annual increase in malaware attacks of 20% for small to large businesses it is surprising to see so many overall website measurement tools having sparse coverage of website security. Fortunately, there a number of specialized Website Security tools. However these tools are themselves fairly mixed in their coverage of Website applications and their prices vary widely. Here are the major options of security services on offer. The table tells the story on how each Website Security tool implements these options:
- Malaware Cleanslng- immediate vs a staged cleaning, removal from blacklists, cost on a fixed rate vs hourly fees;
- Web App Firewall – identify /repel DDoS attacks, brute force login/entry attacks, Bot & zero-day defense, SQL/XSS injection blocks,
- Security Center – Dashboard, continuing scan for malaware hidden at endpoints, scan for Owasp top 10 vulnerabilities, black/white list clients, IPs, Urls, Uptime tracking, Endpoint Detection, Fileless attacks
- Systems Supported – All, PHP CMS, HTML/JS,/CSS, HTML/Java, Enterprise servers;
- Extra Services – CDN-Content Delivery Network, DNS-Domain Name Services, Backup
- Client device vs Server Security scans
|Tool/Measure||Cost||Firewall||Malaware Cleansing||Security Center||Systems||Extras|
|Bitdefender||Pro $250/year 10Clients & 3 servers||Block Brute Force, DDoS, Net exploits||$100 clears PC spyware||Cloud or CSOC EAD & Monitors||All||3 Add-ons|
|CleanTalk||Pro – $104 /year/5 sites||Malaware scan for Net Infections||NA||Cloud||CMS PHP||Comment, registration, spam block|
|cWatch||Pro $95/website||DDoS BadBot blocks.Blacklist,||Fast, free||Cloud CSOC, EDR||CMS PHP. HTML||CDN, DNS|
|OneHourSiteFix||Pro – $65 /year/site||Block DDoS, BadBot , Net Infections||fast, $250 Blacklist Cleanup||Cloud Sharkgate||CMS PHP||Free virus scan|
|Shape||Pro NA||Block BatBot, &Cred Stuffing,||Malaware repair||Cloud Virus Scan, block scraping||All||NA|
|Sitelock||Pro $300/site/year||DDoS BadBot blocking||$200+ Malaware Repair and Hardening||Cloud EDR scan & repair||CMS PHP HTML||DB Tune, Backup,|
|Securi||freemium||DDOS. Blacklist Cleanup, Brute Force Block||6hr $200, Blacklist||Scanner DNS, SSL, checks etc||WP Plugin||CDN, SSL, Free virus scan|
|Wordfence||Pro $100 /site/year||54||16||ClientScan updated b||CMS WP||DB Assist|
- Use both client PC and Website server firewalls. These firewalls detect phishing, download infections, web browsing malaware. The server firewall should also block DDos and Bad Bot attacks,, repair SSL and patch vulnerabilities;
- Have site wide scheduled server-side backups and transaction level triggered backup to Cloud/3rd party especially for eCommerce and multi-user websites;
- Update programs, apps, themes, plugins, libraries, APIs and OS software on server side. [note some hosting servers are responsible for the bulk of these updates]. On client PCs keep OS, browser and integration apps up to date – most often users are responsible for these updates;
- Use sitewide SSL/HTTPS for encrypted messaging to all clients and integrated servers; – this cuts off keylogging, cookie corruption and SSL edits by hackers;
- Use validation procedures to verify all client form input is verified to stymie XSS and SQL injection efforts by hackers;
- Have a cloud or central server CSOC- Cyber Security Operations Center to act as dashboard monitoring the state of a system using EDR-Endpoint Detection & Response plus Antivirus apps tracking & controlling Bad Bot Activity and DDoS -Distributed Denial of Service attacks;
- Use CSOC dashboards to spot fileless scripting attacks and compromised boot/API libraries which can elude EDR systems;
- Actively use Malaware Recovery and Ransomware Remediation methods to repair attacks both on Client PC and local net devices as well as main website servers systems. Expect the repair vendors to accomplish most repairs within 48 hours for about $250 per repair. Expect the repair to include removal of Blacklisting by Google, Facebook , Apple and other web agents;
- Use strong passwords [10-12 chars, 1-3 numbers, 1-3 special case chars, lower & uppercase letters] like this example – Bettor[Passworts](99%0fthetyme]) with no reused passwords, Change them every 3-4 months;
- Have a Cyber Crisis action plan to know what you are going to do in the case of cyber breach, debilitating DDos attack, BadBot breakthrough, ransom-ware incapacitation, malaware infection etc . Also know how to recover from blacklisting by Google or other Cyber-security vendors;
- Never leave a network connected device unattended/unmonitored. as these are targets of Bot agents. Also never decommision a PC, server or database device without purging all network activations and cleansing all associated data storage; and thus closing out Malaware aggregators and competitive scrapers.
- Keep customers, clients and staff informed on a quarterly basis of the latest cyber security alerts and mandates. As a $6 trillion industry , criminal and governmental hacking has plenty of cash and incentives to create new cyber exploits and malaware technology.
Web TrendsFollowing web development and design trends is a major source of information on whats happening on the web. many design studio and development consultancies follow and gauge the latest websites for breakthough opportunities. Here are some examples: Awwwards provides a filtered grid gallery of best design websites sorted by style and technology: Codrops is a playground, collective, and tutorial source for clever Web UI ideas and free code:
Website Analytics & SEOOne of the key drivers and measures for the web has been Website Analytics and the associated SEO Optimizations. And so I have done due diligence and have started to identify 10 Great Google Analytics Alternatives and some key SEO Optimization Tools. But I paused and decided to let others have the pleasure of filling in the latest details on the Web at the Edge of the Darkside. There are 4 reasons for this. First, hacking is not just for criminals and goverments . Now corporates are attacking each other with classic hack attacks as seen in this report from Shape Security: And the list of ever wider cyber security problems is menacing. Second, cookies are portrayed as benign and benevolent web traffic expediters. And for transient cookies that are purged when you exit from a browser this is true. Persistent cookies which are retained in a browser’s cookie library are cited as necessary to retain settings from session to session on a favored website. But persistent cookies may collect and then send a user’s session data back to the visited website without permission or explanation for the use of this private data. Then the users collected data may be shared with 3rd parties . Tracking cookies are the culmination of persistent cookies sleight of hand. In exchange for datae or services supplied to the the visitor, the visitor browsing actions will be made available to 3rd parties with out the visitors permissioin. Tracking cookies are at the shredded edge of privacy. Third, and very telling, tracking cookies plus the open sale of users web actions and data are part of the Google business model. Thus, Google Mail, Google Search, Google Analytics, Android location, and Google Maps data all are available throughout Google as well as being on sale. Yes, all the data have been dutifully anonymized but are available to third parties. The problem is that “anonymized “data can be linked up with numerous secondary sources. with IP address, name, telephone numbers etc gathered through scraping processes or purchases from data breaches and Malaware bots collections. Net result – you see ads suddenly popping up from Funeral Services two days after you mention the need to arrange a funeral . It is no surprise that Pew Research finds nearly 2/3rds of Americans feel they have no control over their web data’s privacy. So it should not be a surprise that Google has faced substantial antitrust and data privacy fines in Europe. The GDPR-General Data Privacy Rules in Europe have partly spread to North America because many websites are implementing the GDPR rules on all their sites . More ominous, Google is now facing bipartisan antitrust actions not just in Congress but in many US State governments. The US Justice Department is planning for broad antitrust action in early October. Given bipartisan support, there is a strong likelihood the way Google and other social media do business will be changed profoundly – privacy rules will likely follow the GDPR template. The net result is how Google Analytics and SEO Optimization work likely will change profoundly over the next year, So expect major changes in how Web Analytics operates in North America.
SEO Savvy ApproachGiven the prospect for major changes in how Web Analyticsand SEO will be done in the near future, we present an approach to SEO that discounts keyword analysis , search position monitoring, or crawling and bots manipulation – rather it supports some key website marketing precepts. Major SEO players such as SearchEngineLand have recogmized a set of SEO Toxins and so are recommending more benign and beneficial methods as seen in their SEO Periodic Table infographic: Thus, some alternative approaches such as external back links from trusted websites, timely updates to tutorial and help notes internal cross links among related website pages, and helpful surveys with timely responses – all of these soft marketing methods get strong promotion. So the following list of tools and consultants emphasize agents who will be able to assist in the with savvy strategies in the face of profound changes to Website Analytics & SEO:
The challenge for website owners and developers is how they adapt to profound changes in the rules on client tracking. Already, the SEO community has shifted away from classic pay for placement and they will find they have to adapt to the new rules of the SEO road.
- Ahrefs [$99/month] has 4 tools for SEO measurements. Site Explorer provide organic search traffic charts and backlink tallies for websites; Site Audit provides per webpage SEO perfomance reults; Rank Tracker reports your sites ranking over time and vis-a-vis your competitors; Content Explorer discovers and analyzes top-performing content in your niche.
- backlinkoo provides SEO insights and consulting well ahead of the curve :
- LinkResearchTools -[399euros] offers tools for Link Building and Monitoring, Link Detoxing, SEO Advisory plus free Chrome & Firefox extensions for link scanning;
- Moz Pro [$79/month] has top rated keyword management, page optimization, and site audit tools;
- NeilPatel sits in the middle with lots of keyword analysis but also Stewide SEO audits;
- Searchmetrics [89euros/month] provides PPC, SERP, & Ranking Research, Competiro,r and SEO Visibility;
- SEMRush [$167/month] has over 40 tools covering keyword management, SEO/PPC/SMM reports, Content Marketing projects, historical data analsis, etc.
- SEOQuake [free]is a Chrome extension acting as a SEO Toolbox with SERP Overlay, Traffic +Backlinks+Ads reports, and SEO Audit.
Website Performance MeasuresThere is a strong consensus among Web Developers and SEO Specialists that your Website’s loading speed is a vital component of your website’s success – equal to brand appeal and clarity plus service messaging strategies. The slider summarizes the key speed measures: [cycloneslider id=”10025″] But as you might expect, the web demands for brand appeal and specalized SEO & service delivery have worked in the past ten years against overall website performance speeds. The problem is the growth in webpage size. – from 700KB/wepage in 2010 to 3500KB /webpage in 2020 a 500% increase, Offsetting the increase in webpage size has bee a number of web system factors like yhr increase in WiFi Internet speed by a factor of 2, faster PC and server system, plus improved browser and server software speeds. But the bottom line is that without careful website performance tuning, heavy and complex page loads will make webpage response time sluggish and slow. Radware tracks the performance of the top 100 eCommerce sites showing the trend to more complex, heavier webpages. The breakout of webpage load growth for these top 100 websites is foretells broader web trends. Here is the data from 2010 to 2020 and it is imposing:
- Image usage doubled from 416KB/webpage to 861KB;
- HTML dropped slightly from 34KB/webpage to 30KB;
- CSS usage doubled from 25KB/webpage to 52KB
- But Font usage ballooned to 105KB/webpage from 2KB in 2010;
- And Video usage skyrocketed from less tan 22KB/webpage to 2735KB.
- Audience Measures delineate key indicators:
- real visitors vs bot scanners or hackers seeking illegal access;
- new vs returning visitors including frequency and duration of visits
- bounce rate – no.of pages visited, duration and page spread [related pages visited],etc.
- Measure Conversions – product/service scans, signups for news letters /bulletins, info downloads, order cart activity including buys or cancels, warranty/support requests, etc;
- Website Overall measures of performance;
- track your website’s uptime and the operational error rate of your website apps;
- measure and record app response time profiles on a daily, weekly, and product cycle basis;
- do biannual assessment of web delivery costs versus conversions and sales.
- adjust offerings and delivery accordingly.
Fortunately there are benign sources of consistent and broad Website KPI. For many companies, their audited monthly and quarterly financial reports provides the starting point for consistent analysis of website systems. For example, the KPI page from the free WP Statistics plugin provides clean website performance data stored locally[hence less privacy comcerns].
There is a small set of WordPress Stats plugins that promise to protect the privacy of visitors data which is collected and use in their website usage reports.
A big advanatge of PageSpeed insights is that it is from Google so it has mobile as well as desktop results. But PageSpeed Insights is sprawling with links to other Google website measures like Google Webmaster Central and Lighthouse. Unlike other tools, users are hunting and clicking around for key data, charts, and recommendations:Although the intention is right, the implementation is awkward.
The only thing missing from free Pingdom is mobile website performance results. But Pingdom allows users to specify where the test are done in the world and allows for easy repeat tests: Here we see the summary performance results [an impressive sub 1 second load time and only 33 page requests and a lean 1.34MB page load of the gallery aided by lazy loading of the images as users scroll down the page]. And immediately blow there are suggested improvements to the page.
But for developers there are the loading charts for all the components – this reveals DNSLT, TTFP, TTI and other critical metrics used in page optimization:
These Pingdom charts may be Greek to the average website owner, but they are vital to Web developers doing the webpage tunings.
WebPageTest is crammed with charts and easily accessed tips – just click on the measure boxes in the upper right; However, like Pingdom, WebPageTest lacks mobile display benchmarking.
This tool is really geared for CDN and international website testing. Users can pick up to simultaneous web test locations for a complete, worldwide picture of how well your website is performing Note how PicsofUSA fails most of the CDN tests because it is currently only target NortAmerican users. But given the Gtranslate plugin that mission may change – and MicroPageTest should provide the insights on how to optimize for an international audience.
GTmetrix free version provides data, timeline charts and advisories for webpage performance:
GTmetrix has tips and tutorial associated with every data result and chart. Each of the page load timings in the above screenshot has a timely tutorial. And GTmetrix Pro adds 3 major features to the program : 1)continuous following of chosen webpages; 2)tracking of more webpages from multiple locations in the world; 3)Developer API allows tracking different display sizes including mobile and tablet displays. The price of the Pro version stretches from $14.95/month to $149.50. Bu as this performance tuning video shows the need for the pro version is not required for most webpage optimization
Uptrends is a paid performance monitoring tool with constant daily monitoring from multiple world-wide locations and for mobile through a variety of device sizes.
Yellow Lab Tools provides webpage results for desktop, tablet and mobile views but does not allow configuration of testing browser or locatiom. Nonetheless, Yellow Lab Tools does provide graphic results and tipsJust click on any metric reading and Yellow Lab pops a full page avisory of what actions might be taken. Its tabletand mobile advisories are valuable.
DareBoost is another example like GTmetrix and UPTrends of programs that offer good webpage testing but have expanded their offerings to complete website performance monitoring. I like the graphic report details:It is easy to see the First Byte, Start Render and FullyLoaded timings . But even more important the number of requests and total weight of the webpage has remained very low after problems with both measures before adding lazy loading, media optimization, and script combination. With DareBoost Pro it is possible to add broader website performance monitoring.
Website Database Performance Measures
Of all the website systems elements, the underlying datadase is the least observed or aapreciated. Partially this comes from the fact that data storage in speed and capacity often available almost as a surplus. But despite this abundance, many web systems have huge BiG Data requirements and fast response tim needs. So website owners should track their database performance.
Here are some of the database monitoring system goals;
- Uptime monitoring as indicated by slow or stalled processes; runaway programs. or crashed apps;
- Monitor changes to the database including any apps or objects added, dropped, or modified;
- Query monitoring for isubstandard operations due to inefficient query plans, nonexistent indexes,clashing apps;
- Tracking resource capacity issues due to unmatched diskspace to CPU, memory or task time;
- Monitor database logs: and hitorical data as key to proactive monitoring, because they contain critical information about where and why systems are under-performing or crashing;
And here are three top end database monitoring tools that can deliver many of the above features:
- SolarWinds Database Performance Analyzer (DPA)
SolarWinds DPA is suitable for SQL Server, Oracle, IBM DB2, SAP ASE , MySQL and many more database engines. It takes a unique approach to database performance monitoring, with a focus on response time and multidimensional performance analysis. This tool ranks database response time as the primary metric.
- dbWatch dbWatch is a Microsoft SQL Server, Sybase, Postgres, MariaDB, MySQL, and Oracle database activity monitoring tool. It has full support for AlwaysOn and RAC clustering, with on-premises, hybrid, or cloud Azure SQL support, making it a truly cross-platform program.
- Opsview Opsview specializes in cloud monitoring tools supporting Oracle, MySQL, InfluxDB, SQL Server, and PostgreSQL Databases. However, Opsview features extensions or addons for monitoring AWS DynamoDB, DB2, Exchange, MariaDB, LDAP, plus capabilities for tracking applications, cloud-based services, networks, servers, and PC clients.
The top end database monitoring tools are geared towards large multi-database and multi-server networked and/or cloud based websites .The tools cost thousands of dollars and require broad training and a range of skills to use them successfully. See the following reviews from Comparitech and DNSStuff for for more details on the trade-offs among the top end DB-Monitoring tools.
Small Business Database Tools
But for the millions of small business websites more appropriate tools are represented by WordPress which currently has 65% market share of the top 10 million CMS websites world wide. A number of SMB website tools like Shopify, Squarespace , Weebly and Wix supply Cloud environs that take care of backups, updates [but not performance tuning, SEO diversity,, external integrations etc] for their users. So we identify some of the top WordPress Database tools. Most are freemium with strong free versions which will meet many WordPress basic database needs with pro options for special rquirements.
- Duplicator[freemium $79/year for 15 sites] scheduled backups, , site cloning and migration – WPE banDuplicator emphasizes its scheduled backups in the Pro edition as much as its migrate and clone capabilities Again, if you choose the Pro edition, it saves to a wide array of Cloud services are. The migrationand clone tool has search and replace feature in the pro edition.
- All-In-One WP-Migration[freemium $79/year for 15 sites] allows both selective backups & migrations. Kin ban
All-in-one WP Migration is one of the easiest plugins to setup for both backups[choice of database and/or themes +plugins and/or special files. For migrations you will need the Pro edition
- Updraft+ [freemium – $57/year for 10 site] Database or full site backup or migration. Kin ban.Updraft offers scheduled database or full website backups to server or remote storage options such as MyDrive, Dropbox, Google Drive, Rackspace Cloud and Amazon S3 web services and its cloud storage 3rdparty apps. Databases can be encrypted prior to backup, Also offers Pro app for full website migration & cloning.
- Boldgrid Total Upkeep[freemium $30/year for unlimited sites] scheduled backups with secure connnections
BoldGrid Total Upkeep offers secure connection to SFTP, Amazon S3, or Googlle Drive for backups which can be database or selected files. Total UpKeep also offers migration features including search and replace option. For $60/year get Total Upkeep, BoldGrid Post & PageBuilder, Easy SEO, 15 Cloud WordPress sites, BoldGrid Total Cache with Speed Coach
- WP-DB Manager [free] cleans and backup database with drop/empty and other SQL queries enabled. WPE ban
- WP-Optimize[freemium $59/year for 5 sites] compresses database and images, caches WP objects. KIN, WPE bansCleans database of selected trash, post/page revisions, ping & trackbacks, spam comments, and defragments MySql table. Schedules clean up scans and auto-compresses new Media Lirary image files after full library compression. Offers caching services with device specific and exclde list rule & triggers browser caching when allowed.
- Participants Database [freemium]allows user to create their own forms and associated database tables
- Query Monitor [free] developers fixed queries into all aspects of WordPress database
Query Monitor is the developer tools panel for WordPress. It enables debugging of database queries, PHP errors, hooks and actions, block editor blocks, enqueued scripts and stylesheets, HTTP API calls, and more.
- Optimizing DB after Erasing Revisions [free] is a ‘One Click’ WordPress Database Cleaner / Optimizer Kin ban
The plugin deletes selected post/page/CPT revision,s spam comments, pimgbacks/trackbacks. ,trash, expired transients, and unused tags. This can be followed by a database optimization.
- ARI Adminer[free]is a phpadmin-like MySQL tables manager. WPE banThis plugin is a delight for developers because they can see any and all of the MySQL tables used by the WP Core , theme, and plugins. Be careful though, misplaced editing of fields and/or tables can cripple your website.
[freemium $59/year/1 site]plugin used
to create tables
& table charts from Excel,
In general these database tools are geared for backend tasks like backup, security, and performance t. The enterprise tools are also geared towards monitoring multi-server networks while tracing changing operational demands. In contrast, the Website SEO & Analytics plus Website Performance tools are are geared to frontend concerns.
All of the tools and measures reviewed here are really designed to cope with the rapid changes and ever increasing complexity of operating on the Web. So like what is happening in Web Analytics expect profound shifts on how things are done on the Web.