Your Personal Web Site

Ways to make your content appear on the web

• GitHub or Pastebin: free, can be anonymous, very few features, you won't collect any ad money. article1, article2
  
  
• Amazon S3 bucket: mostly to learn about AWS, but also can handle high traffic volumes, fast, reliable. article1, article2
  
  
• Facebook page or Pinterest or Trello or Matrix Live etc: free, few features, hard to move to elsewhere, you won't collect any ad money.
  
  
• Tumblr or Medium blog/articles: free, you won't collect any ad money, I don't know the limits of them.
  
  
• Rich-site host: Squarespace or Wordpress or Wix or NodeHost etc: paid, lots of features, maybe lots of support, hard to move to elsewhere, you won't collect any ad money.
  
  
• Medium-site host: Google Sites: free, some features, can use GUI or direct HTML, you won't collect any ad money.
  
  
•   Static site on hosting service: HTML/JS/CSS files hosted on any web hosting service, either hand-built or using design tools.
  
  
• Generated static site on hosting service: write content in Markdown, use a static-site generator (SSG) such as Hugo or Zola or mkws to generate HTML/JS/CSS files for site.
  Kev Quirk's "Static Site Generators Are Easy to Use ... Right?"
  
  
• WYSIWYG-created site on hosting service: write content in a WYSIWYG editor. Froala Vs. CKEditor, BlueGriffon, Wix vs Weebly
  
  
• Web-app site: mix of client HTML/JS/CSS and server engines (PHP, SQL, etc).
  
  
• Business-app site: mix of client HTML/JS/CSS and server engines (PHP, SQL, etc) and gateway to payments/money.
  
  
• VPS and server stack: rent a "virtual machine" (VPS) in the cloud, and install an OS and web server and your pages in it. AWS, DigitalOcean, Linode, etc. Mostly to learn about cloud servers.
  
  
• Local server: a computer on your LAN, and install an OS and web server and your pages on it, and open an incoming port through your router to the server. Need to get a fixed public IP address somehow. Could compromise your LAN's security.
  
  

Parts of your HTML-only site presence

• Domain: name such as "billdietrich.me".
  
  You buy this from a "domain registrar" company, for an annual fee around $20. They associate your domain with your ID, and designate a DNS / nameserver for your domain.
  
  Choose a domain registrar that:
  • Will keep your ID private, for free.
  • Supports 2FA on your domain administration account.
  Recommended: Hover (my referral code), Namecheap.
  
  Ways to have a domain (or sub-domain) without having your name on it:
  Njalla (domain for $15/year)
  FreeDNS (sub-domain for free)
  (but really the middleman, not you, is the owner of the domain)
  
  
  Some TLDs have a blanket bad reputation score; don't get a domain in one of those TLDs. Spamhaus's "The 10 Most Abused Top Level Domains"
  
  
  
• DNS / nameserver: table of records that map from domain to services, such as web host's IP address, email server domain.
  
  Usually you can use the DNS of your domain registar, or the DNS of your web hosting service.
  
  Use the DNS of your web hosting service, so that the DNS records get updated automatically if they move your site to a new IP address.
  
  How to dump your whole DNS configuration (all the records) into a file for backup purposes ?
  
  Tricky: I think DNS has two kinds of records: those that are exposed to public, and those that are not (mainly, DMARC and DKIM).
  
  

  
  # gets some useful info, but not all records:
  nslookup -q=any DOMAINNAME
  
  # gets some useful info, but not all records:
  dig @DNSSERVERNAME ANY -q DOMAINNAME
  
  # about the same:
  dig +nocmd DOMAINNAME any +multiline +answer
  
  # Get specific type of DNS records, using DNS at 1.1.1.1:
  delv @1.1.1.1 DOMAINNAME TXT
  

  
  DNSChecker's "DNS Lookup" also does not give all records; maybe my hosting service is filtering out a bunch of them. For example, it doesn't show my DMARC record, but DNSChecker's "DMARC Validation" does show an evaluation of that record ! Doesn't show DKIM record.
  
  Script "DNSDump.sh" that does a little better (from "How can I list ALL DNS records?"):

  
  #!/bin/bash
  
  # Usage:
  #   DNSDump.sh DOMAINNAME
  #   DNSDump.sh -x DOMAINNAME
  #   DNSDump.sh DOMAINNAME -s NAMESERVER
  
  set -e; set -u
  SUBDOMAINS="www mail mx a.mx smtp pop imap blog en ftp ssh login"
  EXTENDED=""
  
  while :; do case "$1" in
    --) shift; break ;;
    -x) EXTENDED=y; shift ;;
    -s) NS="$2"; shift 2 ;;
    *) break ;;
  esac; done
  DOM="$1"; shift
  TYPE="${1:-any}"
  
  test "${NS:-}" || NS=$(dig +short  SOA "$DOM" | awk '{print $1}')
  test "$NS" && NS="@$NS"
  
  if test "$EXTENDED"; then
    dig +nocmd $NS "$DOM" +noall +answer "$TYPE"
    wild_ips=$(dig +short "$NS" "*.$DOM" "$TYPE" | tr '\n' '|')
    wild_ips="${wild_ips%|}"
    for sub in $SUBDOMAINS; do
      dig +nocmd $NS "$sub.$DOM" +noall +answer "$TYPE"
    done | cat  #grep -vE "${wild_ips}"
    dig +nocmd $NS "*.$DOM" +noall +answer "$TYPE"
  else
    dig +nocmd $NS "$DOM" +noall +answer "$TYPE"
  fi
  

  DNSChecker's "Domain DNS Health Checker Tool"
  MXToolBox
  
  I see comments online saying that "get all records" is called a "zone transfer", and this is not allowed without authorization (an "external zone transfer") because it provides lots of info for attackers.
  
  
  
• Host: machine at the IP address.
  
  
  
• Web server: software listening to ports 80 and 443 on host: Apache, NGINX, others.
  
  
  
• Mail server: software usually listening to ports 25 (SMTP) and 110 (POP3) and 993 (IMAPS) and others on host: Dovecot, Postfix, others.
  
  

How my configuration looks

Users:	Web user	Email user
	HTTP get or post to https://MYDOMAIN.TLD/SOMEPAGE	SMTP/IMAP access to NAME@MYDOMAIN.TLD
Domain registrar:	MYDOMAIN.TLD is owned by MYPERSONNAME Nameservers for MYDOMAIN.TLD are at MYWEBHOST.TLD
	Access DNS at MYWEBHOST.TLD
DNS at Web hosting service:	A record says domain's IPv4 address is n.n.n.n	MX record says mail service is EMAILHOST.TLD DKIM, SPF, DMARC records say where email should come from, what to do about bad email, etc.
	Access n.n.n.n/SOMEPAGE	Access EMAILHOST.TLD
Services:	Web hosting service: Do HTTP service of file n.n.n.n/SOMEPAGE	Email hosting service: NAME@MYDOMAIN.TLD maps to MBOX Do email service using MBOX

Web site hosting choices

Hosting the main site (all pages, maybe all files)

• Paid shared hosting service: best choice, let the pro's handle it, usually comes with free DNS, lots of additional services available (for a price).
  
  Choose a hosting service that:
  • Supports 2FA on your administration accounts.
  • Supports SFTP to upload files to the server.
  
  Examples: Shock Hosting (I use this), NearlyFreeSpeech.net (more barebones).
  
  
• Free shared hosting service: probably limited services, probably no ad money, may have quirks, might not be able to use your own domain name, may go out of business.
  
  
• A server machine in your home LAN: bad choice, you have to open an incoming port through your router into your LAN, monitor security, patch software, etc.
  
  
• A desktop machine in your home LAN: worst choice, now you're letting incoming traffic into a machine with your important personal data on it.
  
  
• Cloud VPS: a virtual Linux machine running in some data center: most flexible, and lets you get hands-on with all the details of web server, database server, etc (if you wish). DigitalOcean. Linode. Security article
  
  

Moving big static files (images etc) to Cloud Storage

This reduces traffic and disk space requirements on your site hosting service.

But this adds another point of failure. If the cloud storage service goes down, your images are inaccessible. Your web pages still work.

Tried Backblaze B2 Cloud Storage.

Requires an email address and phone number. 10 GB free.


KEY LIMIT: cap of 2500 free "downloads" (images displayed, for example) per day. And the only way to eliminate the limit is to add a payment card and get charged for every download above the limit, each day. There is no flat monthly fee or lifetime plan; it's pay-per-download.

Record your "Master Application Key" value from My Settings / App Keys.
Create an application key there too; you will get an "id" value and a "key" value.
Max of 100 buckets, each bucket name must be 6 to 50 chars long, can't rename a bucket.
IMO the only reason to create multiple buckets is if you want different permissions for them.

pip install --upgrade b2
b2 version
b2 help

b2 authorize_account APPIDVALUE
# then when prompted:
Backblaze application key: APPKEYVALUE
# that created file .b2_account_info in your home directory

# you're supposed to get a response giving URL, but I didn't get one
strings .b2_account_info | grep http
# looks like my URL is https://f000.backblazeb2.com/

b2 create_bucket myweb0 allPublic

# bucket has to be empty before you can delete it
# easiest to delete from CLI
b2 delete-bucket BarcelonaPics

b2 list_buckets

# if uploading lots of files, easier to do through web site
#              BUCKET       SRC                  DEST
b2 upload_file myweb0 AndaluciaPics/xxx.jpg AndaluciaPics/xxx.jpg
b2 upload_file myweb0 AndaluciaPics/thumbs/xxx.jpg AndaluciaPics/thumbs/xxx.jpg
# access via: https://f000.backblazeb2.com/file/myweb0/AndaluciaPics/xxx.jpg

"b2 help" puts output on stderr, not stdout ? Can't do "b2 help | less" ?

The CLI utility uploads only one file at a time, and maybe each upload can fail and have to be retried ? Instead, use the web site, go to Browse Files, and in a bucket click Upload. Then you can select N files and upload them in one batch. If the upload fails, just click Close (X) on the error dialog. You will go back to the file-listing dialog (which has no scrollbar or buttons !). Just leave it alone, let it sit there, it will try again on the failed files.

Limited to 2500 file-uploads per day, including retries, and you may well hit that limit when first creating your bucket(s).

Caching content in multiple servers worldwide via a CDN (Content Delivery Network)

You could have a CDN (Content Delivery Network) cache your large data (images, audio, video).

This reduces traffic and disk space requirements on your site hosting service or cloud storage service.

But this adds another point of failure. If the CDN goes down, your images etc are inaccessible. Your web pages still work, because they're addressed via your main domain, not the CDN domain or cloud domain.

Also reduces privacy for your visitors; CDN could track them.

Chandan Kumar's "7 FREE CDN to Speed Up and Secure Your WebSite"
Cloudflare, Akamai, Amazon's CloudFront.
Tim Perry's "Public CDNs Are Useless and Dangerous"

Tried Hostry free plan.

Had to give email, phone, postal address, name of domain of my web site. Verified the email. Later got an email telling me an URL to use, of form like "https://r6t7t9y90e03.hostrycdn.com/..." Then change image links from "https://MYDOMAIN/..." to "https://r6t7t9y90e03.hostrycdn.com/...".

Later decided not to use it. Partly because of additional point of failure, but mainly because I feared it would run up the daily download count on my cloud storage and hit the cap there.

If you use all three (site host, cloud storage, and CDN)

• Pages and small files reside on your site host.
• Big static files (images, videos, etc) reside on the cloud storage.

Then:

• Img links in pages use the CDN's domain.
• CDN is set up to look for images etc on the cloud storage domain.

Your site settings

• Enable HTTPS access to your site, so your users will have secure connections (no malicious injection by ISP).
  
  Apparently, a free TLS certificate generally won't have your person/organization name in it, and just assures that the connection is secure. But a paid TLS certificate generally WILL have your person/organization name in it, and also serves to assure user that they are connecting to correct person/organization.
  
  To see certificate chain from client to your site, on Linux client:

  
  echo | openssl s_client -connect www.YOURDOMAINNAME:443 -showcerts | less
  

  SSL Shopper's "SSL Checker"
  
  
  
• Enable HSTS (HTTP Strict Transport Security).
  
  IT Igloo's "How to configure HTTP Strict Transport Security (HSTS) on Apache & NGINX"
  
  
  
• Implement SubResource Integrity (SRI) if you reference scripts from other sites.
  
  Terence Eden's "Major sites running unauthenticated JavaScript on their payment pages"
  Mozilla's "Subresource Integrity"
  
  
  
• Headers in general:
  
  Max Veytsman's "Everything you need to know about HTTP security headers"
  
  Note: If something works on the local copy of your site ("file" schema) but not on the live hosted copy of your site ("https" schema), it could be due to CSP issues. CSP is not enforced on the local copy of the site.
  
  
  
• Change your port 80 vhost so it doesn't serve your site and only has a .htaccess redirect to port 443, which serves your actual site ?
  
  
  
• For best file caching you should have either an ETag directive or If-Modified-Since / If-None-Match directives, but not both ? And ETag doesn't work if your site is being served from multiple servers.
  AskApache's "Speed Tips: Turn Off ETags"
  
  
  
• See if HTTP/2 (TCP parallel streams and header compression and server push, SPDY) is enabled:
  "sudo curl -v --http2 https://YOURDOMAIN"
  
  
  
• See if HTTP/3 (UDP, QUIC) is enabled:
  "sudo curl -v --http3 https://YOURDOMAIN"
  or
  Chandan Kumar's "How to Test if a Website supports HTTP/3?"
  [To enable HTTP/3 in Firefox, about:config and search for "http3.enabled"]
  
  

Apache .htaccess file

The .htaccess file I have ended up with

Some good things to do in .htaccess file: force use of HTTPS, specify a 404 page, specify security headers, enable GZIP compression.

Some oddities about 404 page: Google will complain that you have a "soft 404", and relative paths will be weird if user tries to access non-existent directory such as "https://YOURDOMAIN/nosuchdir/".

Test validity of .htaccess file

.htaccess check
Security Headers

Adam Michael Wood's "2020 Complete Guide To .htaccess"

/.well-known/security.txt file

Your site should have a file /.well-known/security.txt containing at least:

Contact: mailto:YOUREMAILDADDRESS

IETF draft for security.txt
security.txt
IANA's "Well-Known URIs" (all possibilities)

About page

Your site should have an About page. Maybe it should contain a license statement.
Creative Commons' "Share your work"

Donations

PayPal's "Donate Button"
Katy Teson's "How to Get Donations From 5 Types of Donate Buttons"

Privacy policy page

Probably good to mention some keywords such as cookies, personal, advertising.

Also other warnings and policies that may apply: cookies, GDPR, age restriction, etc.

TermsFeed
FreePrivacyPolicy's "Sample Privacy Policy Template"
WebsitePolicies' "Free Sample Privacy Policy Template"

Sitemap file

Sitemap.xml for search engines:

1. Generate:
   XML-Sitemaps.com
   
   
2. Hand-edit the XML file to (maybe):
   • Remove links to txt/doc/pdf files.
   • Remove links to individual images or special files.
   • Remove any links that have "?" in the URL.
   • Remove all the "lastmod" tags.
   • Remove all the "priority" tags and then add back a few of them for key pages only.
   • Add a "canonical" tag that links to the HTML form of the sitemap.

   
   mv sitemap.xml sitemap.orig.xml
   sed 's/<lastmod>.*<\/lastmod>//' <sitemap.orig.xml \
   | sed 's/<priority>.*<\/priority>//' >sitemap.xml
   

   sitemaps.org's "Sitemaps XML format"
   
   
3. Upload sitemap.xml to your web site.
   
   
4. Validate:
   XML-Sitemaps.com's "XML Sitemap Validator"
   
   
5. Go to Google Search console and add sitemap.xml URL to search.
   
   
6. Go to Bing and add sitemap.xml URL to search.
   
   

I also made a sitemap_canonical.txt file with one URL per line. I forget where I submitted that to.

I also made a hand-made Sitemap.html file for users to use. It doesn't have all of the pages listed, just the main ones.

Google Search Central's "Build and submit a sitemap"
Backlinko's "Sitemaps"
Bing's "Bing Webmaster Guidelines"

robots.txt file

The robots.txt file I have ended up with:

# Sitemap URL must be fully-qualified URL, not relative:
Sitemap: https://www.billdietrich.me/sitemap.xml
# Allow crawling by everything:
User-agent: *
# Disallow would just stop spider, not stop showing in search results.
# So don't disallow anything here, if you wish
# instead use "meta robots noindex" in each page.

Test validity of robots.txt file

Technical SEO's "robots.txt Validator and Testing Tool"
Sitechecker's "Robots.txt Tester"

Google's "Create a robots.txt file"
Yoast's "The ultimate guide to robots.txt"
Yoast's "The ultimate guide to the meta robots tag"

RSS Feed

Kevin Cox's "RSS Feed Best Practises"

Darek Kay's "Style your RSS feed"

Site Structure

Typical structures

• Hierarchical / Tree: home page has categories, each category may contain sub-categories, eventually get to leaf-pages which contain content.
  
  
• Sequential: lead the user through a linear sequence of pages or operations. One example: a blog where the only organization is the date each blog entry was created. Another example: an image-effects site where the sequence is uploadfile-edit-downloadfile.
  
  
• Matrix: similar to "Hierarchical / Tree", but with tags or cross-linking so user can jump across categories to pages in other categories. One example: a real-estate site where user could start by selecting neighborhood, but then select house type and then jump to a listing of those types, maybe in other neighborhoods.
  
  
• Database / Search: user does searches or filters to find pages to go to. One example: an image-search or icon-search site.
  
  

A given site may mix all of these types. For example, Amazon has categories, sequence of shop-cart-checkout, matrix with "similar items" or "users who bought X also bought Y", search. My own site starts with categories, but contains several time-sequential blogs, and has internal links from one page to related section in another page, and has a site-search box.

Your page content

A page has

• Content: words, images, media.
  
  
• Presentation: appearance and layout of the elements.
  
  
• Behavior: what happens when user interacts with the page, or timers cause actions.
  
  

Educated Guesswork's "Web Publishing"

Sharing / re-use / consistency

• Content: iframes for common constructs as page navigation header (section/otherpage images/links) and page navigation footer (About/Home/Search/Privacy links) ? But that's an additional file or two to fetch (cached, I guess, and you can lazy-load) for each page, and how to pass parameters to them ?
  
  [I tried using iframe for a footer, and ran into problems: had to change CSP to allow frame-ancestors:self, but then links opened up inside the iframe area (not full-page), search button failed because references another domain, share button on mobile did nothing. Gave up on it.]
  
  
• Presentation: central CSS file.
  
  
• Behavior: central JavaScript file.
  
  

Standard framework for a page

<!DOCTYPE html>
<html lang="en-US">

<head>
<title>THETITLE</title>
<meta charset="UTF-8">
<meta name="description" content="THEDESCRIPTION">
<meta name="keywords" content="KEYWORD1, KEYWORD2">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" type="text/css" href="YOURSTYLESHEET.css" />
</head>

<body>
<header>STUFF</header>
<h1>THETITLE</h1>
<nav>STUFF</nav>
<main>

<section>
<h2>HEADING</h2>
CONTENT
    <section>
    <h3>HEADING</h3>
    CONTENT
    </section>
CONTENT
</section>

</main>
<footer>STUFF</footer>
</body>

<script type="text/javascript" src="YOURSCRIPTS.js"></script>

</html>

Note that JavaScript file is loaded last, so it doesn't delay display of page.

• Keep it simple, for less maintenance, less chance of breaking, more longevity.
  Jeff Huang's "This Page is Designed to Last"
  
  
• Use an editor that will syntax-check your HTML and CSS and JavaScript. I use VS Code with an HTMLHint extension.
  
  
• Even after the editor says everything is okay, also do browser checks:
  
  In Firefox, open a page, Web Console is crtl-shift-K. See error messages. Fix the ones in your files. A lot of messages may be from files you don't control.
  
  In Chrome/Chromium, open a page, do crtl-shift-I. But it seems you get far less information than in Firefox.
  
  
• Test in several desktop browsers and modes:
  
  
  
  Chrome and Firefox at least.
  
  

  • Check in browser window widths from widest to narrowest.
    
    
  • Check in normal and "dark mode" themes of operating system or browser ?
    
    Adhuham's "A Complete Guide to Dark Mode on the Web"
    MDN's "@media" (see "prefers-color-scheme")
    StackOverflow thread
    Sasha Verbilo's "How to Add a Dark Theme to a Web App" (using invert(100%))
    
    I added "@media (prefers-color-scheme: dark) { ... }" sections to my CSS stylesheet. Also added "color-scheme: light dark;" to ":root".
    
    I struggled with getting dark mode colors to reasonable values. Now the google.com page has a dark mode, so I used that a model. Someone suggested using Contrast Checker. I mostly added statements similar to these in my stylesheet:

    
    @media (prefers-color-scheme: dark) {
      body { background-color: rgb(56, 56, 56); color: white; }
      img { filter: brightness(.8) contrast(1.2); }
      a:link { color: rgb(138, 180, 248); }
      a:visited { color: rgb(138, 150, 248); }
    }
    

    A better way (but it's new): "System Colors"
    
    Can test with Chrome/chromium or Edge browsers. For Firefox, ctrl-shift-I, click Inspector tab, then dark-mode icon.
    
    
  • Try sending the page through Google Translate to another language, and see how it looks there.
    
    
  • Change browser's or OS's main language to something else, and view the page again.
    
    
  • Try printing the page (to PDF or to real printer), and see how it looks there.
    
    In 2020, apparently Firefox doesn't have a print-preview feature, chromium does. I think chromium is automatically removing body and div (but not input) backgrounds when printing, but I'm not sure.
    
    I added an "@media print { ... }" section to the end of my CSS stylesheet to make backgrounds "none".
    
    Hafsa Javed's "How to Force a Page Break in HTML Printing?"
    
    
  • If you're using a lot of third-party scripts and ads, try with/without ad-blocking, tracker-blocking, maybe uBlock Origin extension. At least check that your page still loads with all that stuff blocked.
    
    
  • Check your page with JavaScript disabled in the browser.
    
    
  • Check in "Reader view", if the browser has that.
    In Firefox, Ctrl+Alt+R.
    Apparently there is no way for page code to know it is being viewed in Reader mode.
    
    
  • Windows has a "High Contrast Mode" (WHCM). Test with that.
    Sarah Higley's "Quick Tips for High Contrast Mode"
    
    
  
  
  
• Mobile-friendly:
  
  
  
  Kristen Hicks's "How To Make A Website Mobile Friendly"
  Dmitri Lau's "10 Ways to Make Your Website More Mobile Friendly"
  
  Each IMG tag should have:

  
  <img style="height:NNNpx;width:NNNpx;max-width:100%;" src="..." />
  

  or if say the image is wider but you never want to display it more than 300px wide:

  
  <img style="height:NNNpx;width:NNNpx;max-width:300px;" src="..." />
  

  
  W3Schools' "CSS Media Queries - Examples"
  
  After any site changes, get rid of any cached data before testing
  (on old Android: Settings / Storage & USB / Cached data / Clear;
  on new Android: Settings / Apps & notifications / appname / Storage / Clear Cache).
  
  

  • Test on a real mobile device, in as many browsers as are installed.
    
    
  • Try portrait and landscape orientations.
    
    
  • Try light and dark modes.
    
    
  • If possible, try different screen sizes (phone, tablet, etc).
    
    
  • Some mobile browsers may have the normal view plus a special "desktop site" view; try both.
    
    
  • Anything special about e-readers such as Kindle ?
    
    

  Eruda (give browser devtools in a mobile browser)
  
  
  
  
• Users can Share your site to other places.
  
  Chris Coyier's "On the Web Share API"
  Doesn't work on an offline page.
  2020: works on phone browsers but not on desktop browsers.
  See function window.doSharing() at end of my JS file.
  
  
  
• Advertising.
  
  Google Adsense.
  Make sure your site: has plenty of content, has a privacy policy page that mentions google ads, doesn't use any vulgar language, is mobile-friendly.
  
  In 2020, I gave up on advertising. It compromised user's privacy, didn't make any money for me, and introduced complexity into CSP policy.
  
  
  
• Check for broken links (an endless chore).
  
  If you use VS Code editor, I have written an extension (HTML link checker) that does this from within the editor, where it's easy to fix the code. Can be used to check links in HTML, XML, RSS, PHP, Markdown files.
  
  
  
• Where possible, use HTTPS links instead of HTTP links in your pages, so your users will have secure connections (no malicious injection by ISP).
  
  Use moarTLS browser extension to check your pages for HTTP links. But don't assume that when it says an HTTPS version is available, you can just change the link to HTTPS. Some sites accept HTTPS but give an error or empty page, some give only the home page, some redirect back to the HTTP page.
  
  If you use VS Code editor, I have written an extension (HTML link checker) that does similar checking (and more) from within the editor, where it's easy to fix the code.
  
  
  
• Test for HTML5 compliance.
  
  W3C's "Markup Validation Service" (use More Options and Document Type "HTML5 (experimental)")
  
  Tables / grids:
  
  I think it's considered bad practice or "not HTML5" to use the "table" tag any more, it doesn't adapt well to varying screen sizes ?
  Instead, use "div" or "CSS Grid" ?
  
  CSS Grid:
  w3schools' "CSS Grid Layout Module"
  Surma's "CSS Grid - Table layout is back"
  Rachel Andrew's "Grid by Example"
  But CSS grid doesn't seem any simpler than tables. It adds a few tricks such as being able to automatically re-arrange rows into columns or something.
  
  Divs:
  Tapajyoti Bose's "Grid vs Flex: Where to use which?"
  John Paul Mueller's "Using the Div Tag to Create Tables"
  
  Necati Ozmen's "CSS Grid vs. Flexbox"
  
  Terrific site that can style and generate tables using either table tag or div tags: HTML Div Table Generator. Use the pull-down next to "Table / Div:" to change between the two types. Use "Styler" to choose among various colors and fonts etc. Styler seems to generate only table-tag code; have to run through "convert to div" ? If you have multiple styles of tables in your site, you may have to fiddle with CSS class names to keep them separate.
  
  
  
• Test your CSS files.
  
  W3C's "CSS Validation Service"
  "CSS Stats" (not sure how to use results)
  
  
  
• Test for good coding practices.
  
  Testing in Chrome browser:
  
  In Chrome browser: Go to your page. Click on "..." / More Tools / Developer Tools (or ctrl+shift+I). Click on Audits. Un-check all the Audits except "Best practices". Click the Run Audits button.
  
  Don't submit a form to an explicitly-HTTP URL: Michael Horowitz's "Submitting Data with HTTP and HTTPS"
  
  
  
• Accessible to disabled/impaired people (AKA "a11y"):
  
  
  
  Two kinds of accessibility testing:

  
  Do in this order:
  
  

  1. Static: analyze your HTML code on disk, using editor extensions or other tools. Fix any problems.
     
     
  2. Dynamic: analyze your rendered site pages as they appear in the browser, using browser extensions or other tools. This includes the effects of any server-side or client-side (JavaScript) code that gets executed.
     
     

  
  Static testing in VS Code editor: install extension "Web Accessibility" by Max van der Schee, or "Edge DevTools for VS Code" extension.
  
  Dynamic testing in Firefox browser:

  
  

  1. Preferences / Privacy and Security / "Prevent accessibility services from accessing your browser" must be un-checked.
  2. Open tab with your web page in it.
  3. Click on Hamburger / Web Developer / Accessibility (or press Shift-F12).
  4. Click on "Turn on Accessibility Features".
  5. Look at "Check for issues" status, change pull-down next to it from "None" to "All Issues".
  6. See a table with columns "Role" and "Name".
  7. As you hover the mouse over items in the table, offending elements of your page will be highlighted and a message displayed.

  
  WAVE Browser Extensions
  

  
  Dynamic testing in Chrome/Chromium browser:

  
  "Use the built-in Accessibility audits in the Lighthouse Audits panel in Chrome DevTools, the new accessibility debugging tools in the Elements panel, and/or the aXe extension by Deque Systems."
  Raghavendra Satish Peri's "41 Browser Extensions to Perform Accessibility Testing Effectively"
  
  Chrome browser accessibility audits: Go to your page. Click on "..." / More Tools / Developer Tools (or ctrl+shift+I). Click on Audits. Un-check all the Audits except Accessibility. Click the Run Audits button.
  [Does not work in a local (file-scheme) page.]
  
  Chrome browser Elements: Go to your page. Click on "..." / More Tools / Developer Tools (or ctrl+shift+I). Click on Elements. Halfway down the right-side pane, click on Accessibility. Not sure what to do next, maybe expand the trees and see what is marked "not specified" ?

  
  
  Common accessibility issues:

  
  

  • Don't use just styles (e.g. bold, font size) to make text look like headings. Use the standard heading tags (h1, h2, etc) and then use CSS to style those tags as you wish. This will be better for SEO, too.
    On Linux, show headings of your site:

    
    grep -E '<h[1234]' *.html | sed 's/<h2/\t<h2/' | sed 's/<h3/\t\t<h3/' | sed 's/<h4/\t\t\t<h4/'
    

    SEO Review Tools' "HTML Headings Checker"
    
    
  • Use the semantic <em> and <strong> instead of the visual <i> and <b>.
    
    
  • IMG tag must have ALT attribute. If image is "decorative", just ALT="none".
    
    
  • INPUT tag must be preceded by a LABEL tag, where FOR attribute of LABEL matches ID attribute of INPUT. [Same for BUTTON, TEXTAREA, SELECT tags.]
    
    
  • DIV, SPAN, other tags must have ROLE attribute.
    MDN's "WAI-ARIA Roles"
    
    
  • [As reported by Firefox] Everywhere you have an ONCLICK handler, you also should have an ONKEYDOWN handler ?
    
    
  • [As reported by Chrome] Every page should have a heading, skip link, or landmark region ?
    
    
  • Apparently you can't have a link (A HREF) that contains only an IMG and no text.

  
  
  Issues in standard code from Google:

  
  [As reported by Firefox] Google Search boxes seem to violate the "text label" (LABEL) and "focus styling" requirements.
  
  I tried LABEL FOR="search", LABEL FOR="gsc-i-idl", and entirely enclosing the whole thing in a LABEL tag, but none of them made Firefox happy. The search FORM tag doesn't have an ID attribute, so can't use LABEL FORM="id".
  
  Using CSS was strange:

  
  input[name="search"]:focus { color: red; }
  

  got rid of the "focus styling" violation, but using

  
  input[name="search"]:focus { color: black; }
  

  did not.
  
  [As reported by Firefox] Google Ads seem to violate the "iframe must have title" and "document must have title" requirements, and somehow make Firefox report same about the overall page (even though that does have a title).
  
  [As reported by Chrome] Google Ads seem to violate the "iframe must have title" requirement.
  
  1/2020 Reported the Google issues to Google Disability Support Center. They said they replicated them and engineers will work on them.

  
  Christian Heilmann's "Does your web product support the needs of the many?"
  Matt Stobbs' "The 6 Most Common Accessibility Problems (and How to Fix Them)"
  WebAIM's "Skip Navigation Links"
  Accessible Rich Internet Applications (ARIA)
  

  
  
  
  
  
• Analytics:
  
  Instead of putting analytics code on your pages, and thus sharing user info with some analytics company, see if your site-hosting service already shows that data in their admin interface. Mine does: quite nice reports in CPanel. The downside is that if you change hosting companies, you lose the old data. Also, the data won't show clicks that go to other sites, only accesses to your own pages.
  
  If site-hosting service doesn't do reports but will give you raw data, maybe use: GoAccess
  
  Open-source alternative to Google Analytics: Freshlytics
  Marko Saric's "How to de-Google-ify your site to make it faster and visitor friendly"
  
  
  
• Site Search:
  
  Instead of putting Google Search code on your pages, does your site-hosting service provide a solution ? Apparently mine doesn't.
  
  Christopher Heng's "3 Ways of Putting a Search Engine on Your Website"
  Jennifer Kyrnin's "Adding Search Functionality to Your Website"
  
  FreeFind (displays ads on the search-results page; pay attention to the indexing summaries and logs, tighten down indexing)
  
  Or add an HTML form that goes to DDG:

  
  <form action="https://duckduckgo.com/" method="get">
    <input name="sites" type="hidden" value="www.YOURDOMAIN">
    <input aria-label="Search my site" name="q" type="text" placeholder="Search My Site">
    <button aria-label="Submit search query">&#128270;</button>
  </form>
  

  
  I've been unable to find a search engine that treats headers specially, and gives them as links. For example, if user searches for "ABC" and site has a H1 header with ID attribute and text containing "ABC", I'd expect first search result to be a link to that header's ID. So I've built one: MakeHeaderSearchJS.sh (requires entire header tag is on single line, no double-quotes in text).
  
  
  
• Fonts:
  
  Instead of accessing Google fonts from your pages, and thus sharing user traffic info with Google, just use (user's) system default fonts, or self-host (on your web site) fonts you want to use.
  switching.software's "Bye Google Fonts"
  
  I don't self-host any fonts on my site, or load any from elsewhere in the internet. My CSS lists some fonts to use, in order, but they get used only if the client browser or operating system already has them installed.
  
  Braille Institute's "Atkinson Hyperlegible Font"
  Add "Atkinson Hyperlegible" to list in CSS "font-family" attribute. Maybe put it first in the list: if someone has it installed on their system, they probably did it for a reason.
  
  
  My current font list for most of the page (I don't claim this is good):

  
      font-family: "Atkinson Hyperlegible","Trebuchet MS","Tahoma","Ubuntu","Verdana","Arial","Helvetica","sans-serif";
  

  
  
  

  How fonts are applied to a web page

  
  [I may have some of this wrong.]
  
  Variables in the process:

  
  

  • Your page's CSS could specify an ordered list of font names (e.g. "Courier") and font class names (e.g. "serif"), in a font-family specification for an element.
    (W3Schools' "CSS font-family Property")

    
        font-family: "Trebuchet MS","Tahoma","Ubuntu","Verdana","Arial","Helvetica","sans-serif";
    

    
    
  • Client's browser probably has default-font settings, for various uses (content, menus, buttons, etc).
    
    
  • Client's operating system probably has default-font settings, for various uses (content, menus, window titles, etc).
    
    
  • Client's operating system has a set of fonts installed.
    
    
  • Font files can be residing on your web server, and used from there.
    (W3Schools' "CSS Web Fonts").

    
    @font-face {
      font-family: myFont1;
      src: url(myFont1.woff);
    }
    

    
    
  • Font files can be residing anywhere on the public internet, and used from there.

    
    <link href='http://fonts.googleapis.com/css?family=Rambla|Scada'
        rel='stylesheet' type='text/css'>
    

    
    

  [There is some difference between "fonts" and "font files" which I don't quite understand. I think you process a "font file" to generate "font" images installed in your OS or app ?]

  
  Order in which specifications are checked:

  
  

  1. Definitions/specifications in web page (CSS).
     
     
  2. Definitions/specifications in web browser.
     
     
  3. Definitions/specifications in client operating system.
     
     

  
  Determining font for an element:

  
  

  1. Using the order given in the previous section, get first font name (e.g. "Courier") or font-class name (e.g. "serif").
     
     
  2. If it's a font-class name, using the order given in the previous section, map it to a font name.
     
     
  3. Using the order given in the previous section, find/load the font files for that font.
     
     
  4. If steps 2 + 3 succeeded, we're done, we have the font for the element.
     
     
  5. If anything went wrong in the process (e.g. name unknown, font files not found), go to the next font name in the list and repeat.
     
     

  
  What font is being used for some text on the page ?

  
  In Firefox:
  Ctrl+Shift+I to get into devtools, then on left side click on Inspector tab, then on right side click on Fonts tab. Select an element on the page, and the Fonts pane will show you what font is used. If you select the page's body element, you will see "all fonts used by elements on the page", but no way to find out what element is using some particular font.
  
  In Chrome/chromium:
  Ctrl+Shift+I to get into devtools, then on left side click on Elements tab, then in left pane highlight an element, then on right side click on Styles tab, then in upper-right click on Window icon (hover and it says "Show Computed Styles sidebar"). Scroll to bottom of rightmost pane and see "Rendered Fonts".

  
  MDN's "Web fonts"
  idiomdrottning's "Font size on the Web"
  Aria Beingessner's "Text Rendering Hates You"
  Marcin Wichary's "Typography is impossible"
  
  
  
  
• Navigation / Breadcrumbs:
  
  I use CSS to do a simple side-by-side list.
  Maybe should change it to just a line of text/anchors.
  
  
  
• Syntax-highlighting:
  
  Prism
  Download (for my site): minified, default theme, default languages PLUS "Apache Configuration", "Bash + Shell + Shell", "C", "C++", "CMake", "Java", "JSON + Web App Manifest", "Shell session", "Systemd configuration file", "XML doc (.net)", "YAML".
  
  Change "code" tags to look like:

  
  <pre class="code">
  <code class="language-html" contenteditable="true" spellcheck="false">
  some HTML code
  </code>
  </pre>
  

  
  
  
• Diagrams:
  
  I haven't done these well, yet.
  
  Interesting libraries:
  adrai / flowchart.js (display flowchart)
  Eugene_P's "My Top 13 JavaScript Diagram Libraries"
  Hamza Ed-douibi's "10+ JavaScript libraries to draw your own diagrams"
  Network diagrams:
  xkln's "Creating Network Diagrams with JavaScript" (about Cisco's neXt UI Toolkit)
  
  I tried Cisco's neXt: diagram always appears at end of page, docs seem broken, code is compressed, I think you're limited to one diagram per page ?, last update 4/2016.
  
  
  
• Images:
  
  I pay some attention to how big image files are, but I don't try to compress them down to be ultra-small, for top performance. I do put these attributes on each img tag:

  
  <img style="height:MMMpx;width:NNNpx;max-width:100%;" loading="lazy" src="..." />
  

  
  TinyJPG
  Squoosh
  
  
• Tools to generate static HTML site (from source markdown files):
  
  Hugo
  Alligator.io's "A Brief Tour of the Eleventy Static Site Generator"
  MkDocs (more for documentation, not blog or general web site ?)
  
  From someone on reddit 11/2021:
  
  I use Hugo all the time, I host more than 10 Hugo sites privately and publicly.
  
  I've used Jekyll and Sphinx in the past. Usually I write custom themes for my sites, so my static generator preferences are skewed towards that workflow.
  
  

  • Hugo includes a hot-reload development server, Jekyll and Sphinx do not.
    
    
  • It has no plugins, all features are included in Hugo itself. Features are usually well maintained, unlike many Jekyll plugins.
    
    
  • Go's template language is very well documented, and Hugo's documentation covers pretty much everything Hugo has to offer.
    
    
  • It builds sites very fast. I used to think it was insignificant, but it is very satisfying to have changes up 10 seconds after pushing. Migrating one of my documentation sites from Sphinx to Hugo dropped the CD pipeline duration from 2.5 minutes to 10 seconds.
    
    

  If you need to maintain a static site for a long period of time (over a decade) plan accordingly:
  
  

  • Stick to CommonMark, avoid custom Markdown extensions. Any static generator will be able to render your pages.
    
    
  • Use simple and consistent structure. In my experience a Posts section containing Markdown files works with most static generators and blog themes. Keep the structure consistent to ease writing scripts for migrating to a different structure.
    
    
  • Pick the static generator carefully. You don't want a generator that won't be maintained any more in a couple of years. In my experience Sphinx is the safest bet, it has been around for over a decade and is still very actively maintained. The Python (and Read the Docs) community depends on Sphinx and will likely keep it maintained for quite a while.
    
    

  I would also avoid Next/Gatsby/Nuxt/Gridsome unless you need a very interactive front-end. They add a lot of complexity and are not worth it in my experience if you have less than 5 or so very interactive components.
  
  Mihai Grescenko's "How We Rebuilt Our Documentation Using Hugo"
  
  See "Tried Hugo" section, below.
  
  Site Generators
  JAMstack Themes
  JAMstack == JavaScript, APIs, and Markup.
  
  
  
• Test readability.
  
  WebFX's "Readability Test Tool"
  
  Allen Solomon article about improving readability
  Nida Zehra's "How To Improve Website Readability?"
  
  
  
• Test how fast your pages load.
  
  GTmetrix (numbers are erratic, but recommendations are valuable)
  WebPageTest
  web.dev
  Google's "PageSpeed Insights"
  Pingdom
  "curl -o /dev/null -w 'Conn: %{time_connect} TTFB: %{time_starttransfer} Total: %{time_total} \n' PAGEURL"
  
  Frank van Gemeren's "1001 Questions regarding Web Performance"
  Shivashish Thakur's "10 Effective Tips To Reduce Website & Mobile App Load Time"
  
  
  

Search Engine Optimization (SEO)

Don't try to fix every single thing the tools complain about. Some of them have strange requirements, some requirements contradict each other, and your site doesn't have to get a perfect score. And they probably want you to do things you don't want to do, such as put code from Google and Facebook and Twitter on your pages. But consider each point they make, especially when it comes to titles and descriptions and keywords etc.

"SEO experts agree the best practice is to use one H1 per page. ... If there are multiple H1s, convert them into proper H2-H4s to improve the content's hierarchy." article1, article2

Neil Patel's "SEO Analyzer"
SEO Site Checkup
SEO Tester Online
dareboost
MetricSpot
MetricBuzz
Sitechecker
SEOptimer
Spotibo SEO checker

Jeffrey Vocell's "How to Submit Your Website to Google"
IndexNow
IndexNow.org

Testing in Chrome browser:

In Chrome browser: Go to your page. Click on "..." / More Tools / Developer Tools (or ctrl+shift+I). Click on Audits. Un-check all the Audits except "SEO". Click the Run Audits button.

There are many ways for users to arrive at the home page of your site: they could type "DOMAIN.TLD" or "www.DOMAIN.TLD" or "http://DOMAIN.TLD" or "https://DOMAIN.TLD/index.html" or other things. It might help SEO if you specify the main address for that page ? Add a directive such as:

<link rel="canonical" href="https://www.DOMAIN.TLD/index.html" />

Back-links: apparently if known paid-link-farm sites link to your site, that can drag down your SEO rating ?
Google Search Console (Links / External links)
Derek Gleason's "Google's 'Disavow Links Tool': The Complete Guide"
MozRank Checker (also useful for finding errors)
Ubersuggest
RankWatch

W.S. Toh's "11 Simple HTML Tips For On-Page SEO"
W.S. Toh's "6 Places to Add Keywords To An HTML Website"

You can put a "Twitter card" on your home page, or on key pages, or on every page:
Courtney Seiter's "The Everything Guide to Twitter Cards"
Card validator (have to log in to Twitter to use it)

You can put "OpenGraph" info on your home page, or on key pages, or on every page:
Michal Pecanek's "Open Graph Meta Tags"
Preview and Generate Open Graph Meta Tags
Adam Coti's "The Essential Meta Tags for Social Media"
Facebook's "Sharing Debugger" (have to log in to Facebook to use it)

You can put "JSON Linked Data" info on your home page, or on key pages, or on every page:
Google's "Understand how structured data works"
JSON for Linking Data
JSON-LD Examples
Google's "Structured Data Testing Tool"

Yet another schema: Dublin Core Metadata Element Set:
Wikipedia's "Dublin Core"

Yet another tag: ISSN:
Terence Eden's "How to add ISSN metadata to a web page"

HTML5 Microdata

Lets you mark common objects in your pages.

Some useful types for me: Person, Movie, Blog, Book, Guide, HowTo, AboutPage, SoftwareApplication, ComputerLanguage, TouristTrip, City/Country/State, TouristDestination, Product ?

schema.org's "Getting started with schema.org using Microdata"
schema.org's "Full Hierarchy"
MDN's "Microdata"

Google's "Structured Data Testing Tool"
Microdata.reveal by jcdarwin (Chrome)
OpenLink Structured Data Sniffer (Firefox)

I mostly gave up on using microdata on my pages. It just added a lot of complications and extra tags for little discernible benefit.

You'll probably know you've done SEO and site-map well if you start getting spam from companies wanting to place articles on your site or help you improve your presence on search-engines or make your site better on mobile ! You're showing up on their RADAR somehow.

I know my site is ugly

I know my site has big pages, clunky icons, doesn't look modern.

I'm much more focused on content, instead of form. It would be nice to have a better UI. But given a choice, instead of UI I'd spend the time on more content or fixing broken links.

I've struggled with the design for a couple of decades now.

Some things I like about the current design:

• Big pages are easier to search through than a lot of small pages.
• Layout adapts to mobile / smaller screens tolerably well.
• Search engines easily understand the pages.
• Easy to copy info out of the pages.
• Works even with JavaScript disabled.
• Site seems to work with any browser. [Mainly because of previous item.]
• Everything is client-side, no back-end services (database, PHP, etc) needed.
• Site easily portable to a new hosting service. [Mainly because of previous item.]
• Site works locally via "file://" URL. [Again because no back-end.]
• Site should be highly accessible to sight-impaired people.
• Checking for broken links and fixing them is straightforward.
• No fancy menus or link styles etc: just standard, basic, well-known UI elements.
• I want to keep learning more about HTML/CSS/JS, for development and bug-bounty purposes, so don't want some framework that would have me just writing in Markdown.
• Mobile: My pages are large and start with an index of sections and links to other pages. I see no gain from hiding those under a Hamburger menu icon.

Some things that used to be more important to me, when I lived on a boat and wrote for boaters:

• Big pages are easier to save to disk and view offline than a lot of small pages.
• Fairly easy to print the pages.
• Pages load fast because not many big images on most of them.

Seirdy's "An opinionated list of best practices for textual websites"

Photo gallery pages

digiKam app has Tools / Generate HTML Gallery menu item.

Single File PHP Gallery (sfpg).

Python script:

#!/usr/bin/env python3
import glob
print ('<ul id="nav">')
for link in glob.glob('*'):
   print ('  <li><a href="{0}">{0}</a></li>'.format(link))
print ('</ul>')

I created some custom files and shell scripts for my site, to make pages that use lozad.js:
MiscPics/MakeGalleryHTML-Head.txt
MiscPics/MakeGalleryHTML-Tail.txt
MiscPics/MakeGalleryHTML.sh
Each directory of image files has a sub-directory "thumbs" containing thumbnail images.
This gives a page such as MiscPics.html

I tried Hugo, and it was a fiasco

Migrate existing site to Hugo

Hugo's "Migrate to Hugo"
But nothing for HTML to Hugo.

By default, in Hugo your page content is written in Markdown. HTML is allowed (Hugo's "Supported Content Formats"), but Markdown seems to be the standard (and for more tools than just Hugo; JAMstack), and I want my site to work better on mobile. So I need to convert my existing HTML files to Markdown.
pandoc
Available in Mint's Software Manager, but it's an old version, install from home.

pandoc -f html -t markdown myfile.html >myfile.md

Directory Structure

Hugo's "Directory Structure"
Hugo's "Content Organization"

Source files (*.md) go in archetypes/ directory (config variable "archetypeDir") ? But also content/ directory (config variable "contentDir") and data/ directory (config variable "dataDir") ? Seems content/ is for your page files.

"In Hugo, the simplest way to create a page is to create a directory containing an index.md file. If you need to create a new section for your website, then instead of an index.md file, simply create _index.md. Section pages in Hugo are mostly called list and other pages are called single." from Syna

Default Hugo target directory for your built website is public/ (config variable "publishDir").

Hugo's "Configure Hugo"

Simplest to use a single config.toml file in the root directory, if you're not going to do multiple languages or development/staging/production things. But the theme you choose may come with a lot of features, including multiple languages ?

By default, the static/ directory in the site project is used for all static files (e.g. stylesheets, JavaScript, images). The static files are served on the site root path ...
Hugo's "Static Files"

My directory structure:

archetypes
content
    sailing
        gettingstarted
		gettingboat
		usingboat
	    places
        magnolia
	    	blog
    computers
    reasoning
    legal
    places
        europe
	    	romaniatrip2016blog
	    	romaniatrip2019blog
	spain
		andalucia
	    	blog
		barcelona
	    	blog
    cartalk
    disasterprep
    presidents
data
layouts
static
    pics1
    pics2
    pics3
themes
    hugo-theme-w3css-basic

hugo new site ./mywebsite
cd mywebsite

Themes

hugo's "Hugo Themes"
How to pick a theme ? Look for features you want, I guess.

I want: blog, image gallery, deeplinks, syntax highlighting, mobile. I'd like something that doesn't use lots of dependencies. I think my first choice is Hugo Themes / W3.CSS Basic, maybe second choice would be Hugo Themes / Swift.

# to get ALL themes:
git clone --recursive https://github.com/spf13/hugoThemes ./themes
# to get ONE theme:
mkdir themes
cd themes
git clone https://github.com/jaden/twentyfourteen

hugo new SECTIONNAME/FILENAME.FORMAT
# will create file content/SECTIONNAME/FILENAME.FORMAT
# but then no html file is produced in public ???

# Generate site's pages:
rm -fr public
hugo

sudo iptables -I INPUT -p tcp --dport 1313 -j ACCEPT
hugo server
browse to http://localhost:1313

I installed from gohugoio / hugo

I installed hugo_0.58.2_Linux-64bit.deb. "Extended" version adds Sass/SCSS support; I think this means it can run on a VPS such as DigitalOcean or Linode or AWS.

Installer put it in /usr/local/bin/hugo. Do "hugo version" to verify.

Tried Hugo Themes / W3.CSS Basic:

# make site "myweb1" in home directory
cd ~
git clone https://github.com/it-gro/hugo-theme-w3css-basic temp1
cd temp1
mv exampleSite ../myweb1
mkdir ../myweb1/themes
mkdir ../myweb1/themes/hugo-theme-w3css-basic
mv * ../myweb1/themes/hugo-theme-w3css-basic
cd ..
rm -fr temp1

cd myweb1
# edit config.toml:
# change values of baseURL and title
# change I want:
uglyURLs = true
# addition I want:
disablePathToLower = true

cd content
# create index.md file containing any text

cd ..
rm -fr public
hugo
# see that public was created and contains files
ls public

hugo server
# browse to http://localhost:1313
# Site home page appeared !  But I don't see contents of my index.md file.
# close browser tab
ctrl-C to kill server

# Edit config.toml to disable jumbotrons, clients, testimonials,
# photocards, params.marquee, params.features, cookieconsent, params.seeMore
# Comment out setting of disqusShortname to disable it.
# Set disableRSS to true.

rm -fr public
hugo
hugo server
# browse to http://localhost:1313
# Now should see contents of my index.md file.

# Use "preview w3.css color themes" pull-down menu to try different color schemes.
# I like "2017 island-paradise" and "Light blue #87ceeb"
# In config.toml, comment out line for "2018-ceylon-yellow.css"
# and un-comment line for "light-blue.css".
# Maybe you could make your own color-scheme by going to
# themes/hugo-theme-w3css-basic/static/vendor/w3css/4
# and copying and modifying one of the files.
# Docs say "You may create your own monochromatic color theme using
# https://www.w3schools.com/w3css/w3css_color_generator.asp
# and then put the css file under /static/css/w3-theme-custom.css".
# Once you're happy with the color-scheme, you can remove the pull-down
# by changing themeColorSelectorEnable to false.

# add image gallery
https://github.com/liwenyip/hugo-easy-gallery/
content/pages/showcase/gallery-01.md

# STUCK !!! Can't get any *.md file I add under content/ to produce
# a corresponding *.html file under public/.

Tried Hugo Themes / Swift:

# make site "myweb3" in home directory
cd ~
hugo new site ./myweb3
cd myweb3
cd themes
git clone https://github.com/onweru/hugo-swift-theme.git

cd ..
hugo new Reason/Reasoning.md
# edit content/Reason/Reasoning.md

rm -fr public
hugo
# got errors "found no layout file for "HTML" for "page": You should create a template file ..."
# edited config.toml to add line 'theme = "hugo-swift-theme"'

rm -fr public
hugo
# got error "Transformation failed: TOCSS: failed to transform "scss/main.scss"
# (text/x-scss): this feature is not available in your current Hugo version"
# Apparently this theme is using SCSS, only available in Hugo extended.

# So, I downloaded the "extended" deb file and installed over top of existing Hugo.
# Maybe could have done "snap refresh hugo --channel=extended" instead ?
rm -fr public
hugo

hugo server
# browse to http://localhost:1313
# Site works, I see home page and my added Reasoning.md file

# copied menu.yml file from themes/hugo-swift-theme/exampleSite/data/ to top data/

# added more files in content/Reasoning folder, and they all show up as
# tiles/cards/posts on the home page, not what I want.

hugo new Presidents/Presidents.md
# also shows up as a tile on home page, not what I want.
# I can't find how this is happening.

Tried Hugo Themes / Academic:

# make site "myweb4" in home directory
cd ~
hugo new site ./myweb4
cd myweb4
cd themes
git clone https://github.com/gcushen/hugo-academic.git

cd ..
git init
git submodule update --init --recursive

rm -fr public
hugo
# errors, lots of stuff missing

# tried copying everything up from exampleSite to main
# directory, got different errors

# edited config.toml to add line 'theme = "hugo-academic"'

# edit config/_default/params.toml to set "color_theme" as desired

# https://sourcethemes.com/academic/docs/get-started/

Tried Hugo ReFresh:

# make site "myweb5" in home directory
cd ~
hugo new site ./myweb5 && cd myweb5
git init
git submodule add https://github.com/PippoRJ/hugo-refresh.git themes/hugo-refresh
rm config.toml
curl -O https://raw.githubusercontent.com/PippoRJ/hugo-refresh/master/exampleSite/config.yaml

rm -fr public
hugo server -D

# edit config.yaml file

# site works, but again I don't see where index.html is being generated,
# don't see how to change it to a page of icons as I want.

I started from a handcrafted HTML web site, with inline JavaScript on some pages, and Google ads.

Hugo processes images to make the site; I think I want to disable that.

My evaluation of Hugo

I've tried 4 different themes. None of them document the content structure you're supposed to use and what they do with it, the required filenames. None of them define the limits they impose, the assumptions they make.

With W3.CSS Basic, I couldn't get any of my pages to appear. Put *.md files all over various levels of content/, nothing.

With Swift, all my *.md files end up as "cards" on the home page. Not documented, didn't expect that, I guess no way to change it.

With Academic, got error after error while building. Couldn't get to a sane site.

With ReFresh, don't see how to modify home page to what I want, it's hard-coded somewhere ? Similar problem in other themes.

And it looks like I would have to structure pages totally differently for some themes ? Syna, for example, defines "fragments". Some use "widgets". In some of them, every one of my pages has to be in a separate directory, and named _index.md ? So once I start using a theme, I'm stuck with it, no easy way to move to another theme or port my site to some other framework.

Since each theme is different and seems to involve a fair amount of custom code, I'm not sure how much the general Hugo community can help with big problems. With small "what is the syntax for this element ?" questions, they probably could help. But for "why don't any of my pages appear ?", you'll have to ask the community (if any) specific to that particular theme. And some of the themes I tried seem to have an unresponsive developer and no community; not sure.

I've read tons of documents, looked through lots of the code, and I'm totally defeated. This is crazy.

Realized I tried something that was way too complex for my needs. Would be nice to have canned things such as photo gallery, blog, nice UI. But not at this cost.

Periodically check your site

• Connection and header tests:
  
  Qualys SSL Labs' "SSL Server Test"
  dataskydd.net's "How privacy-friendly is your site?"
  Mozilla's "Observatory"
  Mozilla's "Laboratory (Content Security Policy / CSP Toolkit)" add-on
  CSP Evaluator
  Security Header Scanner
  ImmuniWeb's "Website Security Test"
  crt.sh's "Certificate Search"
  KeyCDN's "HTTP/2 Test"
  Connect via HTTP/2 ? "curl --http2 -I https://YOURDOMAIN/"
  Connect via HTTP/3 ? "curl --http3 -I https://YOURDOMAIN/"
  HTTP/3 Check
  
  Note: If something works on the local copy of your site ("file" schema) but not on the live hosted copy of your site ("https" schema), it could be due to CSP issues. CSP is not enforced on the local copy of the site.
  
  
  
• Test validity of .htaccess file:
  
  See Test validity of .htaccess file section.
  
  
  
• Test validity of robots.txt file:
  
  See Test validity of robots.txt file section.
  
  
  
• Test validity of HTML:
  
  W3C Markup Validation Service
  Use Linux's "tidy" command. E.g. "tidy -errors -quiet *.html >/dev/null"
  See DoTidy.sh
  
  
  
• Test/view DNS information:
  
  ViewDNS.info
  Manytools
  Online-Domain-Tools
  
  
  
• Speed test:
  
  Google's "PageSpeed Insights"
  
  
  
• Test how web pages look on various devices:
  
  
  • MobileTest.me
    Had to comment out setting of X-Frame-Options to 'deny' in my site's .htaccess file.
    Got it working in Chromium, but not in Firefox.
    It doesn't replicate a weird button-sizing issue someone saw with my site on an iPhone.
    
    
  • mobiReady
    
    
  • Responsively
    
    
  • In Chrome or Chromium browsers:
    Go to web page, then F12 for Developer's Tools. Click Toggle Device Toolbar button (looks like two overlapping rectangles) or Ctrl-Shift-M to turn Device Mode on or off (icon is blue when on). Select device with the Viewport Controls (pull-down menu that starts as "Responsive"). Refresh web page. Every time you change device type, refresh page again.
    
    
  
  
• Test for tracking:
  
  Blacklight
  
  
  
• Test reputation:
  
  Scambunkers
  Scamvoid
  Securi
  Lenny Zeltser's "Tools for Looking up Potentially Malicious Websites"
  Simple Email Reputation
  Test the Spammyness of your Emails
  
  Gonzalo Sainz Trapaga's "A new and innovative way for Google to kill your SaaS startup"
  
  
  
• Spell-check:
  
  "sudo zypper install codespell"
  "codespell *.html"
  
  
  

Moved my web site to Arch Hosting, just to change to SSL

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.billdietrich.me/$1 [R,L]

Header set Strict-Transport-Security "max-age=31536000" includeSubDomains env=HTTPS

Header set Strict-Transport-Security "max-age=31536000; includeSubDomains" env=HTTPS

Moved my web site to Shock Hosting

Header set Content-Security-Policy "script-src 'self' 'unsafe-inline' lots of Google domains;"
Header set Content-Security-Policy "style-src 'self' 'unsafe-inline' lots of Google domains;"

# script hashes:
#   contents of myscripts.js == 'sha256-something1'
Header set Content-Security-Policy "script-src 'self' 'sha256-something1' lots of Google domains;"
Header set Content-Security-Policy "style-src 'self' 'unsafe-inline' lots of Google domains;"

Header set Content-Security-Policy "default-src 'unsafe-inline' 'unsafe-eval' *;"

Created my wife's web site on Wix

Miscellaneous

Clipboard

User click copies text to clipboard (site or app must have "clipboard-write" permission ?):

<span onclick="navigator.clipboard.writeText('This text will be copied')">Click here to copy</span>

User click copies text from clipboard (site or app must have "clipboard-read" permission ?):

<span onclick="navigator.clipboard.readText()
  .then(clipText => document.querySelector('.cliptext').innerText = clipText);
  ">Get contents of clipboard</span><br />
<span class="cliptext">contents of clipboard</span><br />

SoByte's "Clipboard Operations"

Time-wasting files

You could put attractive nonsense files on your site to waste the time of bots or attackers.

Some people say this will be ineffective; bots are looking for very-low-hanging fruit such as vulnerable Wordpress plug-ins and plaintext credentials and open RDP ports.

Check with your hosting service to make sure this doesn't violate their TOS. My hosting service said: I can't host anything malicious, and a zip-bomb file qualifies as "malicious", not allowed.


Dr. Neal Krawetz's "It's All Fun and Games"

David Fifield's "A better zip bomb"
Make sure this won't blow up anti-virus on your site host.

"Give a 2 MB dump from /dev/rand that is AES-encrypted with a long random password."
AES-256 password is limited to 32 chars.
Secret Messages' "Encrypt Files"
You get a ZIP file with an AES file inside it.
UNZIP it to get the .aes file, set basename to something enticing, put on your site.

Give a password manager database that has a good master password and only junk contents. Important that it has a well-known file extension, so scanners can find it. Or (a lot of work) have the contents look like real account entries, but be bogus.

Give fake AWS credentials files:

sudo apt install awscli
aws --version

aws configure
# type AWS Access Key ID such as AAAABBBBCCCCDDDDEEEE
# type AWS Secret Access Key ID such as AAAABBBBCCCCDDDDEEEE
# type default region name such as us-west-2
# choose "default output format"
cat ~/.aws/config
cat ~/.aws/credentials

aws ec2 create-security-group --group-name "n" --description "d"
# should get authentication failure

# copy config and credentials files to /.aws on your web site

Give fake Kubernetes Secrets file:

echo -n 'admin' | base64      # USERNAME
echo -n 'PASSWORD' | base64   # PASSWORD

Then create a file "secret.yaml" containing:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: USERNAME
  password: PASSWORD

And put that file on the web site.

Another Kubernetes thing ? Make a directory ".kube" with a file "config" or "sa-config" in it ?

apiVersion: v1
kind: Config
users:
- name: svcs-acct-dply
  user:
    token: SOMETHINGHERE
clusters:
- cluster:
    certificate-authority-data: SOMETHINGHERE
    server: 192.168.1.1
  name: self-hosted-cluster
contexts:
- context:
    cluster: self-hosted-cluster
    user: svcs-acct-dply
  name: svcs-acct-context
current-context: svcs-acct-context

Give a ZIP file with a name such as "mypasswords.zip" that has a good master password and only junk contents:

zip -e mypasswords1.zip FILENAMES
zip -e mypasswords.zip mypasswords1.zip
rm mypasswords1.zip

VeraCrypt containers have no fixed header or standard file extension, so don't give a fake VC container, the attacker won't know what it is.

But: LUKS containers DO have a fixed header format. So maybe create an empty container file and put it on the site. See LUKS encryption section of my Linux Storage page.

Fake Linux /etc/shadow file: copy your real /etc/shadow file from your home PC, then edit to copy your normal-user encrypted password field to root's entry, then change same 10-20 chars in the middle of each encrypted password to random stuff.

Fake Linux /etc/passwd file: copy your real home PC /etc/passwd file, then make sure no personal info is in it. No need to edit it further; it has no encrypted passwords, and doesn't apply to the site host server anyway.

Fake Linux "pass" files:

sudo apt install gnupg2
sudo apt install pass

# https://fedoraproject.org/wiki/Creating_GPG_Keys
# https://www.fossmint.com/pass-commandline-password-manager-for-linux/

pass init SOMEUSERNAME@YOURDOMAINNAME
# creates ~/.password-store

pass generate FBUSERNAME/facebook.com 30
# failed on my system; something broken in gpg
# maybe because I uninstalled seahorse ?

pass insert FBUSERNAME/facebook.com
# and type a password, twice
# failed on my system; something broken in gpg
# maybe because I uninstalled seahorse ?

pass ls

Then copy the whole .password-store tree to your web site.

Crypto wallet.
Install Electrum app, get passphrase and set password, File / Save Copy of wallet, get it from ~/snap/electrum/2/.electrum/wallets/ directory, also do Wallet / Private Keys / Export, get /snap/electrum/2/electrum-private-keys.csv, put them in /.bitcoin directory on web server.
https://linuxconfig.org/how-to-install-monero-wallet-on-linux-gui-cli

Make sure the files are accessible to public. Also a good idea to allow directory listing to public.

Maybe create a password-protected directory, if the hosting service supports this:

The following is not best practice, but may be the best you can do on a shared hosting service.

For Apache server: Create a directory (with attractive name) you want to protect, and in that directory create a .htaccess file containing:

AuthType Basic
AuthName "Restricted Content"   # this will be displayed to user before login
AuthUserFile /.htpasswd-DIRNAME
Require valid-user

Use CLI command "htpasswd" (or 4WebHelp or Alterlinks ) to create a username:encyptedpassword string, put it into file /.htpasswd-DIRNAME, then set permissions to 600 or 640. (Best practice would be to put that file outside the site tree, but you can't do that on a shared host. Also you could name it ".htRANDOMCHARS" if you didn't want scanners to see it's a password file.)

Go to a browser and try to access "YOURSITE/DIRNAME". You should get prompted for username and password.

I wasn't able to get it working fully; I think the AuthUserFile path is wrong. But I left the site with the .htaccess file present and no .htpasswd file anywhere, and that works fine for my purpose: password-protected directory that's impossible to log into.

Make your site look like it's running Wordpress when it's not:

Create directories wp-content/plugins, wp-content/themes, wp-includes/js, wp-admin, wp-uploads.

Create files wp-includes/wlwmanifest.xml (maybe copy https://wpsmackdown.com/wp-includes/wlwmanifest.xml), license.txt (maybe copy https://wpsmackdown.com/license.txt), wp-links-opml.php (maybe copy and edit output of https://wpsmackdown.com/wp-links-opml.php), xmlrpc.php, wp-admin/wp-login.php, wp-admin/install.php, wp-config-sample.php, wp-config.php, readme.html

Tricky: Since my web server doesn't actual execute PHP, there is no harm in putting PHP files on my site. And the wp-links-opml.php file I put there is actually the output of the execution of a typical wp-links-opml.php file. Same for xmlrpc.php file.

Actually, later I found that my server DOES execute PHP. So I created a wp-login.php file that gives a delay and then a plausible login-failed page:

<!DOCTYPE html>
<html>
<head>
<title>Login</title>
</head>
<body>
<?php sleep(10.0); ?>
<?php echo '<p>ERROR: Incorrect username or password.</p>'; ?>
<!-- ?php phpinfo(); ? -->
</body>
</html>

I couldn't get it to redirect to a large file, or set HTTP response code. Apparently you have to do those before sending any output, and even then I couldn't get them to work.

Add to pages:

<meta name="generator" content="WordPress 3.0" />
<!-- This site is optimized with the Yoast SEO plugin v12.5 - https://yoast.com/wordpress/plugins/seo/ -->
<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="https://s0.wp.com/wp-includes/wlwmanifest.xml" />

<img src="https://s0.wp.com/wp-content/themes/vip/99u-2016/assets/img/logo-white-adobe-hover.svg" alt="adobe" width="0" height="0" style="visibility:hidden" />

Install themes: Go to https://wordpress.org/themes/ , select a theme, click Download button on it, download ZIP file, save ZIP file into wp-content/themes. Extract contents, producing directory THEMENAME and files inside it. Delete ZIP file. Go into THEMENAME directory and delete all files except index.php and style.css . On your site, create directory wp-content/themes/THEMENAME and upload index.php and style.css to it.

To get a known-vulnerable theme, I took 1.2.2 from https://github.com/themegrill/flash/releases

Use https://whatcms.org/ and https://cmsdetect.com/ and https://scanwp.net/ and https://www.wpthemedetector.com/ and https://whatwpthemeisthat.com/ to test the site.

https://www.billdietrich.me/index.html
curl --data "param1=value1&param2=value2" https://wpsmackdown.com/xmlrpc.php

Thorsten Frommen's "How to Detect WordPress Websites"

Apparently some bots look for favicon files to detect login pages. They try port 80 or 443 and fetch /favicon.ico, or fetch page and parse '<link rel="icon"' out of it to get favicon path. See if icon's hash matches the hash of one they want (Wordpress, some control panel, etc).

Fake Microsoft Dynamics 365 Commerce .env file: .env