Ways to make your content appear on the web

The more you depend on some company's site or external software, the less work you have to do, but at a greater risk that: they could go out of business, or close your account, or censor you, or make changes that you don't like, or the software could break when something else changes.

Also, the more you do by hand, the more you will learn about the guts of HTTP, HTML, CSS, JavaScript, web servers. All valuable skills.

But as my hand-built site gets bigger, making the same infrastructure change (such as, changing footer) on 200 pages gets pretty painful. Maybe there's a way to use "sed" to do batch changes.

Honestly Undefined's "Blogging vs. blog setups" (cartoon)

Can you have one web site which is accessible through both clearnet and onion (Tor) ? See Tor Server section of my Connection Security page.

Rest of this page will mostly focus on the "Static site on hosting service" case.

Parts of your HTML-only site presence

Don't have the same company be your domain registrar and your site host. If the site host disables your account for some reason, you want to be able to point your domain to some other host and get the site back online quickly.

Do not use a free email account provided by your domain registrar or your site host. If they disable your account for some reason, you lose both email and web site.

How my configuration looks:

Users: Web user Email user
HTTP get or post to https://MYDOMAIN.TLD/SOMEPAGE
Domain registrar: MYDOMAIN.TLD is owned by MYPERSONNAME

Nameservers for MYDOMAIN.TLD are at MYWEBHOST.TLD
DNS at Web hosting service: A record says domain's IPv4 address is n.n.n.n MX record says mail service is EMAILHOST.TLD

DKIM, SPF, DMARC records say where email should come from, what to do about bad email, etc.
Access n.n.n.n/SOMEPAGE
Services: Web hosting service:

Do HTTP service of file n.n.n.n/SOMEPAGE
Email hosting service:


Do email service using MBOX

Web site hosting choices

Hosting the main site (all pages, maybe all files):

  • Paid shared hosting service: best choice, let the pro's handle it, usually comes with free DNS, lots of additional services available (for a price).

    Choose a hosting service that:
    • Supports 2FA on your administration accounts.
    • Supports SFTP to upload files to the server.

  • Free shared hosting service: probably limited services, probably no ad money, may have quirks, might not be able to use your own domain name, may go out of business.

  • A server machine in your home LAN: bad choice, you have to open an incoming port through your router into your LAN, monitor security, patch software, etc.

  • A desktop machine in your home LAN: worst choice, now you're letting incoming traffic into a machine with your important personal data on it.

  • Cloud VPS: a virtual Linux machine running in some data center: most flexible, and lets you get hands-on with all the details of web server, database server, etc (if you wish). DigitalOcean. Linode. Security article

Moving big static files (images etc) to Cloud Storage:

This reduces traffic and disk space requirements on your site hosting service.

But this adds another point of failure. If the cloud storage service goes down, your images are inaccessible. Your web pages still work.

Tried Backblaze B2 Cloud Storage.
Requires an email address and phone number. 10 GB free.

KEY LIMIT: cap of 2500 free "downloads" (images displayed, for example) per day. And the only way to eliminate the limit is to add a payment card and get charged for every download above the limit, each day. There is no flat monthly fee or lifetime plan; it's pay-per-download.

Record your "Master Application Key" value from My Settings / App Keys.
Create an application key there too; you will get an "id" value and a "key" value.
Max of 100 buckets, each bucket name must be 6 to 50 chars long, can't rename a bucket.
IMO the only reason to create multiple buckets is if you want different permissions for them.

pip install --upgrade b2
b2 version
b2 help

b2 authorize_account APPIDVALUE
# then when prompted:
Backblaze application key: APPKEYVALUE
# that created file .b2_account_info in your home directory

# you're supposed to get a response giving URL, but I didn't get one
strings .b2_account_info | grep http
# looks like my URL is https://f000.backblazeb2.com/

b2 create_bucket myweb0 allPublic

# bucket has to be empty before you can delete it
# easiest to delete from CLI
b2 delete-bucket BarcelonaPics

b2 list_buckets

# if uploading lots of files, easier to do through web site
#              BUCKET       SRC                  DEST
b2 upload_file myweb0 AndaluciaPics/xxx.jpg AndaluciaPics/xxx.jpg
b2 upload_file myweb0 AndaluciaPics/thumbs/xxx.jpg AndaluciaPics/thumbs/xxx.jpg
# access via: https://f000.backblazeb2.com/file/myweb0/AndaluciaPics/xxx.jpg
"b2 help" puts output on stderr, not stdout ? Can't do "b2 help | less" ?

The CLI utility uploads only one file at a time, and maybe each upload can fail and have to be retried ? Instead, use the web site, go to Browse Files, and in a bucket click Upload. Then you can select N files and upload them in one batch. If the upload fails, just click Close (X) on the error dialog. You will go back to the file-listing dialog (which has no scrollbar or buttons !). Just leave it alone, let it sit there, it will try again on the failed files.

Limited to 2500 file-uploads per day, including retries, and you may well hit that limit when first creating your bucket(s).

Caching content in multiple servers worldwide via a CDN (Content Delivery Network):

You could have a CDN (Content Delivery Network) cache your large data (images, audio, video).

This reduces traffic and disk space requirements on your site hosting service or cloud storage service.

But this adds another point of failure. If the CDN goes down, your images etc are inaccessible. Your web pages still work, because they're addressed via your main domain, not the CDN domain or cloud domain.

Chandan Kumar's "7 FREE CDN to Speed Up and Secure Your WebSite"
Cloudflare, Akamai, Amazon's CloudFront.

Tried Hostry free plan.
Had to give email, phone, postal address, name of domain of my web site. Verified the email. Later got an email telling me an URL to use, of form like "https://r6t7t9y90e03.hostrycdn.com/..." Then change image links from "https://MYDOMAIN/..." to "https://r6t7t9y90e03.hostrycdn.com/...".

Later decided not to use it. Partly because of additional point of failure, but mainly because I feared it would run up the daily download count on my cloud storage and hit the cap there.

If you use all three (site host, cloud storage, and CDN):

  • Pages and small files reside on your site host.
  • Big static files (images, videos, etc) reside on the cloud storage.
  • Img links in pages use the CDN's domain.
  • CDN is set up to look for images etc on the cloud storage domain.

Your site settings

Apache .htaccess file

The .htaccess file I have ended up with:

ErrorDocument 404 /404.html

# https://www.digitalocean.com/community/tutorials/how-to-rewrite-urls-with-mod_rewrite-for-apache-on-ubuntu-16-04
# https://httpd.apache.org/docs/2.4/rewrite/flags.html
# https://httpd.apache.org/docs/current/mod/mod_rewrite.html
RewriteEngine On
# change HTTP to HTTPS:
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.billdietrich.me/$1 [R=301,L]
# change no-WWW to WWW:
RewriteCond "%{HTTP_HOST}"   "!^www\.billdietrich\.me"
RewriteRule ^(.*)$ https://www.billdietrich.me/$1 [R=301,L]

Header set Strict-Transport-Security "max-age=31536000; includeSubDomains" env=HTTPS

# causes a bug that makes indexing fail; appends with comma when it should use semicolon
#Header append Content-Type charset=UTF-8

Header set Pragma "public;"
Header set Cache-Control "public;"

Header Set X-Robots-Tag "all;"

Header set Content-Security-Policy "frame-ancestors 'none'; default-src 'self'; object-src 'self'; img-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';"

Header set X-Frame-Options "deny"
Header set X-Content-Type-Options "nosniff"
Header set X-XSS-Protection "1; mode=block"

# I don't set "same-origin" on Referrer-Policy.
Header set Referrer-Policy "no-referrer-when-downgrade"
Header set Feature-Policy "payment 'none'; microphone 'none'; camera 'none'"
Header set Permissions-Policy "payment=(), microphone=(), camera=()"
Header set Expect-CT "enforce,max-age=600"

Header set Access-Control-Allow-Origin "https://www.billdietrich.me"
Header set Vary "Origin"

ServerSignature Off
Header always unset "X-Powered-By"
Header unset "X-Powered-By"
Header always unset "Server"
Header unset "Server"

<IfModule mod_expires.c>
  ExpiresActive On

  # Images
  ExpiresByType image/jpeg "access plus 1 year"
  ExpiresByType image/gif "access plus 1 year"
  ExpiresByType image/png "access plus 1 year"
  ExpiresByType image/webp "access plus 1 year"
  ExpiresByType image/svg+xml "access plus 1 year"
  ExpiresByType image/x-icon "access plus 1 year"

  # Audio / Video
  ExpiresByType audio/mp43 "access plus 1 year"
  ExpiresByType video/mp4 "access plus 1 year"
  ExpiresByType video/mpeg "access plus 1 year"

  # CSS, JavaScript
  ExpiresByType text/css "access plus 1 week"
  ExpiresByType text/javascript "access plus 1 week"
  ExpiresByType application/javascript "access plus 1 week"

  # Others
  ExpiresByType application/pdf "access plus 1 year"

Test validity of .htaccess file

Adam Michael Wood's "2020 Complete Guide To .htaccess"

/.well-known/security.txt file

Your site should have a file /.well-known/security.txt containing at least:

Contact: mailto:admin@billdietrich.me
IETF draft for security.txt

About page

Your site should have an About page. Maybe it should contain a license statement.
Creative Commons' "Share your work"


Privacy policy page

Also other warnings and policies that may apply: cookies, GDPR, age restriction, etc.

FreePrivacyPolicy's "Sample Privacy Policy Template"
WebsitePolicies' "Free Sample Privacy Policy Template"

Sitemap file

Sitemap.xml for search engines:
  1. Generate:

  2. Hand-edit the XML file to (maybe):
    • Remove links to txt/doc/pdf files.
    • Remove links to individual images or special files.
    • Remove any links that have "?" in the URL.
    • Remove all the "lastmod" tags.
    • Remove all the "priority" tags and then add back a few of them for key pages only.
    mv sitemap.xml sitemap.orig.xml
    sed 's/<lastmod>.*<\/lastmod>//' <sitemap.orig.xml \
    | sed 's/<priority>.*<\/priority>//' >sitemap.xml
    sitemaps.org's "Sitemaps XML format"

  3. Upload sitemap.xml to your web site.

  4. Validate:
    XML-Sitemaps.com's "XML Sitemap Validator"

  5. Go to Google Search console and add sitemap.xml URL to search.

  6. Go to Bing and add sitemap.xml URL to search.

robots.txt file

The robots.txt file I have ended up with:

# Sitemap URL must be fully-qualified URL, not relative:
Sitemap: https://www.billdietrich.me/sitemap.xml
# Allow crawling by everything:
User-agent: *
# Disallow would just stop spider, not stop showing up in search results.
# So don't disallow anything here, instead use "meta robots noindex" in each page.

Test validity of robots.txt file

Google's "Create a robots.txt file"
Yoast's "The ultimate guide to robots.txt"
Yoast's "The ultimate guide to the meta robots tag"

Your page content

A page has:

  • Content: words, images, media.

  • Presentation: appearance and layout of the elements.

  • Behavior: what happens when user interacts with the page.

Sharing / re-use / consistency:

  • Content: iframes for common constructs as page navigation header (section/otherpage images/links) and page navigation footer (About/Home/Search/Privacy links) ? But that's an additional file or two to fetch (cached, I guess, and you can lazy-load) for each page, and how to pass parameters to them ?

    [I tried using iframe for a footer, and ran into problems: had to change CSP to allow frame-ancestors:self, but then links opened up inside the iframe area (not full-page), search button failed because references another domain, share button on mobile did nothing. Gave up on it.]

  • Presentation: central CSS file.

  • Behavior: central Javascript file.

Standard framework for a page:


<!DOCTYPE html>
<html lang="en-US">

<meta charset="UTF-8">
<meta name="description" content="THEDESCRIPTION">
<meta name="keywords" content="KEYWORD1, KEYWORD2">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" type="text/css" href="YOURSTYLESHEET.css" />
<script type="text/javascript" src="YOURSCRIPTS.js"></script>





Search Engine Optimization (SEO)

Don't try to fix every single thing the tools complain about. Some of them have strange requirements, some requirements contradict each other, and your site doesn't have to get a perfect score. And they probably want you to do things you don't want to do, such as put code from Google and Facebook and Twitter on your pages. But consider each point they make, especially when it comes to titles and descriptions and keywords etc.

Neil Patel's "SEO Analyzer"
SEO Site Checkup
SEO Tester Online
Spotibo SEO checker

Jeffrey Vocell's "How to Submit Your Website to Google"

Testing in Chrome browser:
In Chrome browser: Go to your page. Click on "..." / More Tools / Developer Tools (or ctrl+shift+I). Click on Audits. Un-check all the Audits except "SEO". Click the Run Audits button.

There are many ways for users to arrive at the home page of your site: they could type "DOMAIN.TLD" or "www.DOMAIN.TLD" or "http://DOMAIN.TLD" or "https://DOMAIN.TLD/index.html" or other things. It might help SEO if you specify the main address for that page ? Add a directive such as:

<link rel="canonical" href="https://www.DOMAIN.TLD/index.html" />

Back-links: apparently if known paid-link-farm sites link to your site, that can drag down your SEO rating ?
Google Search Console (Links / External links)
Derek Gleason's "Google's 'Disavow Links Tool': The Complete Guide"
MozRank Checker (also useful for finding errors)

W.S. Toh's "11 Simple HTML Tips For On-Page SEO"
W.S. Toh's "6 Places to Add Keywords To An HTML Website"

You can put a "Twitter card" on your home page, or on key pages, or on every page:
Courtney Seiter's "The Everything Guide to Twitter Cards"
Card validator (have to log in to Twitter to use it)

You can put "OpenGraph" info on your home page, or on key pages, or on every page:
Michal Pecanek's "Open Graph Meta Tags"
Preview and Generate Open Graph Meta Tags
Adam Coti's "The Essential Meta Tags for Social Media"
Facebook's "Sharing Debugger" (have to log in to Facebook to use it)

You can put "JSON Linked Data" info on your home page, or on key pages, or on every page:
Google's "Understand how structured data works"
JSON for Linking Data
JSON-LD Examples
Google's "Structured Data Testing Tool"

Yet another schema: Dublin Core Metadata Element Set:
Wikipedia's "Dublin Core"

HTML5 Microdata

Lets you mark common objects in your pages.

Some useful types for me: Person, Movie, Blog, Book, Guide, HowTo, AboutPage, SoftwareApplication, ComputerLanguage, TouristTrip, City/Country/State, TouristDestination, Product ?

schema.org's "Getting started with schema.org using Microdata"
schema.org's "Full Hierarchy"
MDN's "Microdata"

Google's "Structured Data Testing Tool"
Microdata.reveal by jcdarwin (Chrome)
OpenLink Structured Data Sniffer (Firefox)

I mostly gave up on using microdata on my pages. It just added a lot of complications and extra tags for little discernable benefit.

If security is an issue (you're handling money, storing personal data, etc), use as few plug-ins (e.g. Wordpress plug-ins) as possible. Each is a potential source of vulnerabilities.

Don't allow comments or user-uploaded content; they're just a source of grief. Moderation of them is a headache. And they even could trigger blacklisting of your site. Just give an email address. Or, if you want to develop a community, create a reddit sub for your site/brand. comments section is a hive of scum and villainy
Third-party scripts (for ads, search, analytics, comments, trackers, widgets, etc) can be very dangerous, even doing key-logging (AKA "session recording") while users use the web page (article 1, article 2). You may be using a script for one purpose (say, comments section), but it may pull in 10 other scripts that do all kinds of things. Test your site with Blacklight.

Coding HTML, CSS, JavaScript:
"Web Skills"
Devslopes' "Full Stack Web Development Tutorial Course" (131 videos, watch the ones you wish)
CSS-Tricks' "A Complete Guide to CSS Media Queries"
1-Line Layouts (maybe only in latest browsers)

W.S. Toh's "Simple Responsive Pure CSS Hamburger Menu"

The CSS3 Test (tests your browser)

Grab colors/fonts from a good-looking site: "Colors and Fonts Extractor" by nh2amine (Firefox extension)
Tricky: Lea Verou's "CSS3 Patterns Gallery"
Totally insane: Chris Pattle's "The Simpsons in CSS"

Don't specify font-sizes (and maybe element spacings ?) in pixels; that doesn't adjust as user changes browser font-size. Specify in percents, qualifiers such as "larger", or rems (the font-size of the root <html> element). article1, article2

I know my site is ugly:

I know my site has big pages, clunky icons, doesn't look modern.

I'm much more focused on content, instead of form. It would be nice to have a better UI. But given a choice, instead of UI I'd spend the time on more content or fixing broken links.

I've struggled with the design for a couple of decades now.

Some things I like about the current design:
  • Big pages are easier to search through than a lot of small pages.
  • Layout adapts to mobile / smaller screens tolerably well.
  • Search engines easily understand the pages.
  • Easy to copy info out of the pages.
  • Works even with Javascript disabled.
  • Site seems to work with any browser. [Mainly because of previous item.]
  • Everything is client-side, no back-end services (database, PHP, etc) needed.
  • Site easily portable to a new hosting service. [Mainly because of previous item.]
  • Site works locally via "file://" URL, again because no back-end services.
  • Site should be highly accessible to sight-impaired people.
  • Checking for broken links and fixing them is straightforward.
  • No fancy menus or link styles etc: just standard, basic, well-known UI elements.
  • I want to keep learning more about HTML/CSS/JS, for development and bug-bounty purposes, so don't want some framework that would have me just writing in Markdown.
  • Mobile: My pages are large and start with an index of sections and links to other pages. I see no gain from hiding those under a Hamburger menu icon.
Some things that used to be more important to me, when I lived on a boat and wrote for boaters:
  • Big pages are easier to save to disk and view offline than a lot of small pages.
  • Fairly easy to print the pages.
  • Pages load fast because not many big images on most of them.

Seirdy's "An opinionated list of best practices for textual websites"

I tried Hugo, and it was a fiasco

Later, I heard that much depends on what theme you choose: anyone can make a theme, and some are not well-built or well-documented. But I don't know which are the best.

Install from snap or gohugoio / hugo

Migrate existing site to Hugo:

Hugo's "Migrate to Hugo"
But nothing for HTML to Hugo.

By default, in Hugo your page content is written in Markdown. HTML is allowed (Hugo's "Supported Content Formats"), but Markdown seems to be the standard (and for more tools than just Hugo; JAMstack), and I want my site to work better on mobile. So I need to convert my existing HTML files to Markdown.
Available in Mint's Software Manager, but it's an old version, install from home.

pandoc -f html -t markdown myfile.html >myfile.md

Directory Structure:

Hugo's "Directory Structure"
Hugo's "Content Organization"

Source files (*.md) go in archetypes/ directory (config variable "archetypeDir") ? But also content/ directory (config variable "contentDir") and data/ directory (config variable "dataDir") ? Seems content/ is for your page files.

"In Hugo, the simplest way to create a page is to create a directory containing an index.md file. If you need to create a new section for your website, then instead of an index.md file, simply create _index.md. Section pages in Hugo are mostly called list and other pages are called single." from Syna

Default Hugo target directory for your built website is public/ (config variable "publishDir").

Hugo's "Configure Hugo"

Simplest to use a single config.toml file in the root directory, if you're not going to do multiple languages or development/staging/production things. But the theme you choose may come with a lot of features, including multiple languages ?

By default, the static/ directory in the site project is used for all static files (e.g. stylesheets, JavaScript, images). The static files are served on the site root path ...
Hugo's "Static Files"

My directory structure:

Hugo's "Quick Start"

hugo new site ./mywebsite
cd mywebsite


hugo's "Hugo Themes"
How to pick a theme ? Look for features you want, I guess.

I want: blog, image gallery, deeplinks, syntax highlighting, mobile. I'd like something that doesn't use lots of dependencies. I think my first choice is Hugo Themes / W3.CSS Basic, maybe second choice would be Hugo Themes / Swift.

# to get ALL themes:
git clone --recursive https://github.com/spf13/hugoThemes ./themes
# to get ONE theme:
mkdir themes
cd themes
git clone https://github.com/jaden/twentyfourteen

Create pages:

What is a section ?

# will create file content/SECTIONNAME/FILENAME.FORMAT
# but then no html file is produced in public ???

# Generate site's pages:
rm -fr public

Test your site locally:

sudo iptables -I INPUT -p tcp --dport 1313 -j ACCEPT
hugo server
browse to http://localhost:1313

Community: Hugo forums

VSCode extensions: Hugo Language and Syntax Support, Hugo Snippets.

I installed from gohugoio / hugo:

I installed hugo_0.58.2_Linux-64bit.deb. "Extended" version adds Sass/SCSS support; I think this means it can run on a VPS such as DigitalOcean or Linode or AWS.

Installer put it in /usr/local/bin/hugo. Do "hugo version" to verify.

Tried Hugo Themes / W3.CSS Basic:

# make site "myweb1" in home directory
cd ~
git clone https://github.com/it-gro/hugo-theme-w3css-basic temp1
cd temp1
mv exampleSite ../myweb1
mkdir ../myweb1/themes
mkdir ../myweb1/themes/hugo-theme-w3css-basic
mv * ../myweb1/themes/hugo-theme-w3css-basic
cd ..
rm -fr temp1

cd myweb1
# edit config.toml:
# change values of baseURL and title
# change I want:
uglyURLs = true
# addition I want:
disablePathToLower = true

cd content
# create index.md file containing any text

cd ..
rm -fr public
# see that public was created and contains files
ls public

hugo server
# browse to http://localhost:1313
# Site home page appeared !  But I don't see contents of my index.md file.
# close browser tab
ctrl-C to kill server

# Edit config.toml to disable jumbotrons, clients, testimonials,
# photocards, params.marquee, params.features, cookieconsent, params.seeMore
# Comment out setting of disqusShortname to disable it.
# Set disableRSS to true.

rm -fr public
hugo server
# browse to http://localhost:1313
# Now should see contents of my index.md file.

# Use "preview w3.css color themes" pull-down menu to try different color schemes.
# I like "2017 island-paradise" and "Light blue #87ceeb"
# In config.toml, comment out line for "2018-ceylon-yellow.css"
# and un-comment line for "light-blue.css".
# Maybe you could make your own color-scheme by going to
# themes/hugo-theme-w3css-basic/static/vendor/w3css/4
# and copying and modifying one of the files.
# Docs say "You may create your own monochromatic color theme using
# https://www.w3schools.com/w3css/w3css_color_generator.asp
# and then put the css file under /static/css/w3-theme-custom.css".
# Once you're happy with the color-scheme, you can remove the pull-down
# by changing themeColorSelectorEnable to false.

# add image gallery

# STUCK !!! Can't get any *.md file I add under content/ to produce
# a corresponding *.html file under public/.

Tried Hugo Themes / Swift:

# make site "myweb3" in home directory
cd ~
hugo new site ./myweb3
cd myweb3
cd themes
git clone https://github.com/onweru/hugo-swift-theme.git

cd ..
hugo new Reason/Reasoning.md
# edit content/Reason/Reasoning.md

rm -fr public
# got errors "found no layout file for "HTML" for "page": You should create a template file ..."
# edited config.toml to add line 'theme = "hugo-swift-theme"'

rm -fr public
# got error "Transformation failed: TOCSS: failed to transform "scss/main.scss"
# (text/x-scss): this feature is not available in your current Hugo version"
# Apparently this theme is using SCSS, only available in Hugo extended.

# So, I downloaded the "extended" deb file and installed over top of existing Hugo.
# Maybe could have done "snap refresh hugo --channel=extended" instead ?
rm -fr public

hugo server
# browse to http://localhost:1313
# Site works, I see home page and my added Reasoning.md file

# copied menu.yml file from themes/hugo-swift-theme/exampleSite/data/ to top data/

# added more files in content/Reasoning folder, and they all show up as
# tiles/cards/posts on the home page, not what I want.

hugo new Presidents/Presidents.md
# also shows up as a tile on home page, not what I want.
# I can't find how this is happening.

Tried Hugo Themes / Academic:

# make site "myweb4" in home directory
cd ~
hugo new site ./myweb4
cd myweb4
cd themes
git clone https://github.com/gcushen/hugo-academic.git

cd ..
git init
git submodule update --init --recursive

rm -fr public
# errors, lots of stuff missing

# tried copying everything up from exampleSite to main
# directory, got different errors

# edited config.toml to add line 'theme = "hugo-academic"'

# edit config/_default/params.toml to set "color_theme" as desired

# https://sourcethemes.com/academic/docs/get-started/

Tried Hugo ReFresh:

# make site "myweb5" in home directory
cd ~
hugo new site ./myweb5 && cd myweb5
git init
git submodule add https://github.com/PippoRJ/hugo-refresh.git themes/hugo-refresh
rm config.toml
curl -O https://raw.githubusercontent.com/PippoRJ/hugo-refresh/master/exampleSite/config.yaml

rm -fr public
hugo server -D

# edit config.yaml file

# site works, but again I don't see where index.html is being generated,
# don't see how to change it to a page of icons as I want.

I started from a handcrafted HTML web site, with inline JavaScript on some pages, and Google ads.

Hugo processes images to make the site; I think I want to disable that.

My evaluation of Hugo:

I've tried 4 different themes. None of them document the content structure you're supposed to use and what they do with it, the required filenames. None of them define the limits they impose, the assumptions they make.

With W3.CSS Basic, I couldn't get any of my pages to appear. Put *.md files all over various levels of content/, nothing.

With Swift, all my *.md files end up as "cards" on the home page. Not documented, didn't expect that, I guess no way to change it.

With Academic, got error after error while building. Couldn't get to a sane site.

With ReFresh, don't see how to modify home page to what I want, it's hard-coded somewhere ? Similar problem in other themes.

And it looks like I would have to structure pages totally differently for some themes ? Syna, for example, defines "fragments". Some use "widgets". In some of them, every one of my pages has to be in a separate directory, and named _index.md ? So once I start using a theme, I'm stuck with it, no easy way to move to another theme or port my site to some other framework.

Since each theme is different and seems to involve a fair amount of custom code, I'm not sure how much the general Hugo community can help with big problems. With small "what is the syntax for this element ?" questions, they probably could help. But for "why don't any of my pages appear ?", you'll have to ask the community (if any) specific to that particular theme. And some of the themes I tried seem to have an unresponsive developer and no community; not sure.

I've read tons of documents, looked through lots of the code, and I'm totally defeated. This is crazy.

Realized I tried something that was way too complex for my needs. Would be nice to have canned things such as photo gallery, blog, nice UI. But not at this cost.

Karan Sharma's "Migrating my blog to Zola" (some commentary about Hugo)

From Rogerio Brito's "Alternatives to ikiwiki?":
I have tried to use hugo, but configuring it is totally crazy: first, you have to decide on a theme and, then, (almost?) everything that you do is tied to that theme. This is (almost) the opposite to the philosophy of LaTeX: first, write your text and, only then, worry about the style/looks. Separation of content and form doesn't seem to be the priority, from a few days looking at it.


That's not to mention the myriad of hugo themes and theme authors that try to bribe you into using their hosted solutions (despite branding everything as "open source! OMG!"), like "wowchemy" - you will have a hard time untangling the instructions of their so-bloated themes to be usable on your local computer; so much so to the point that you give up with their convoluted configuration (which, potentially, doesn't "transfer" to other themes, if you are worried about possibly changing themes in the future).

Periodically check your site

Moved my web site to Arch Hosting, just to change to SSL

5 March 2018, I bought a lifetime-hosting subscription from Arch Hosting for $25.

Terms are 2 GB storage and 500 GB/month bandwidth. No monthly or annual hosting charge, for life. If you want to buy a domain through them, first year is free, subsequent years cost somewhat above market rate (so they can make a profit).

If you want a free-first-year domain from them, there is a list of about 8 TLD's it can be in. My TLD, ".me", is not one of them.

I already had a domain registered with GoDaddy, so I selected "I will use my existing domain and update my nameservers".

Ended up with three sets of username/password for Arch: one for the account, one for the control panel, and one for FTP.

Had to wipe out files on my old host before switching domain to point to new host, because only access I had to old host was via FTP. That means my site will be down for several hours during the changeover. Maybe I could have done FTP to old site later by using IP address instead of domain name, but I didn't think of that.

Switched domain's IP and nameserver IPs in GoDaddy to point to Arch's servers, and suddenly GoDaddy says "we can't show you anything about DNS because now you're on someone else's DNS". Maybe that wouldn't have happened if I'd changed only the domain's IP and left the nameserver IPs unchanged (which I think would have worked, but not been optimal). Now my DNS info is accessed through Zone Editor in Arch's cPanel. Domain still is registered at GoDaddy.

Took about an hour for the updated DNS info to percolate through the system; GoDaddy had a TTL of 1 hour on it.

In Arch file manager, site files must reside under /public_html folder. If you want to FTP directly into there, when creating FTP account, specify home directory "public_html".

Couldn't figure out how to get the SFTP to work, used FTP instead, it worked fine. I'm using a client (WinSCP) that they don't support, so none of their supplied config files help me. Maybe I need SSH enabled on my site in order to use SFTP. And maybe I need an encryption key, too. [Eventually asked Support, they said yes need SSH, no key, they'll enable SSH on my account. Login details a bit different, but now SFTP works.]

About 24 hours later, got email that Arch's "AutoSSL" had generated an SSL certificate for me (from Let's Encrypt) and installed it, all automatically with no request by me. Went to browser, and both HTTP and HTTPS work.

When I go to HTTP site, HTTPS Everywhere add-on does not automatically send me to HTTPS site. Turns out HTTPS Everywhere works off a set of rules, it doesn't automatically switch over for every random site.

Info in browser says the certificate expires in 3 months ? Answer from Support: "The SSL certificate expires in 3 months, but the system automatically renews it before that. As long as your site points to our hosting, the certificate will always renew before the expiration date."

Site test gets an "A" rating from Qualys SSL Labs' "SSL Server Test".

Received an automated "you're close to your disk space limit" email. Looks like the limit is 1.95 GB, not 2.0, and I've used 1.62 GB. Also a limit of 100K files, and I have about 9K.

Ran moarTLS on HTTP version of my site, and it correctly detects that an HTTPS version is available. Change to HTTPS version, and moarTLS says all internal links are secure.

Added some Apache "Rewrite" directives to ".htaccess" file, and now any access to my site using HTTP gets changed to HTTPS automatically.

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.billdietrich.me/$1 [R,L]
I wonder if I should delete some superfluous CNAME records in my DNS entry. I haven't enabled any additional features: email, etc. But my site is accessible through both "www.mydomain.tld" and "mail.mydomain.tld", and probably other sub-domains too. Wikipedia's "List of DNS record types"

There's no "AAAA" record in my DNS entry, so I think my site is not accessible via IPv6. ipv6 test and chair6 confirm that.

By end of 2nd day, I've interacted with Support on about 6 questions, with good results each time. Very satisfied. And my site is completely up and running, no issues remaining. Of course, my site is pretty simple, no server-side code, not using their email or Wordpress or database or other features.

Looks like I can upgrade to 10 GB storage and 1 TB/month bandwidth for $20. This would change my lifetime account from "Startup" to "Business".

After a week, everything still fine. Traffic to my site has consumed about 4 GB of bandwidth. So I won't be anywhere near hitting the 500 GB/month limit.

After 2+ weeks, received email invoice saying "Pay $0.00". Followed by another email saying "Thank you for your payment of $0.00".

A month after that email invoice, received another pair of emails, same thing. I guess it will happen every month.

11 days later, got ANOTHER email invoice/paid pair.

6/16: Noticed that HSTS is not enabled on my site. Support says add

Header set Strict-Transport-Security "max-age=31536000" includeSubDomains env=HTTPS
to the .htaccess file. Did that and it worked. Later changed to

Header set Strict-Transport-Security "max-age=31536000; includeSubDomains" env=HTTPS

7/1: No DNSSEC on my domain; I tested it with VeriSign's "DNSSEC Analyzer". But my domain was registered through GoDaddy; not sure if that matters, I think Arch's DNS servers are serving it now.
Arch Hosting Wiki's "DNSSEC" says DNSSEC is optional, sounds a little scary, may cause problems.

8/29: Found that Arch offers 2FA (software TOTP) on the cPanel login, so I enabled that. They're hoping to offer 2FA on the main account login in the future.

8/29: Arch doesn't support onion (Tor) access to your web site, and doesn't plan to do so, opposes it.

6/1/2019: Realized my site had a DNS MX record, and a default email account in the hosting account. Support says they are harmless and can't be removed.

8/2019: cPanel greatly raised charges, so Arch moved users (who consented) to DirectAdmin. They said nothing would change with FTP, but in fact SSH and SFTP went away, only plain FTP is available now.

10/2019: Set MX records to point to Migadu servers. Note: MX records should contain domain names, not IP addresses.

10/2019: Found you get a limited number of Support credits when you sign up, and mine have run out, apparently no Support for me any more !

11/2019: Now TOTP is supported again on main Arch account login, but TOTP on DirectAdmin fails for me, says time-discrepancy error.

There is a "restricted community" /u/ArchHosting on reddit. I don't know how that differs from a "restricted sub-reddit", which would start with /r/.

12/2019: Suddenly got a notice that Arch Hosting is going out of business, in 1 month ! They have arranged for easy migration and special pricing to Shock Hosting, which seems to have unlimited storage (nice). But probably the end for my $25 lifetime deal.

Moved my web site to Shock Hosting

Shock Hosting

Paid about $21/year for hosting, because I'm getting a huge discount arranged by Arch.

2019-12-15: Got the new hosting account active, after a bit of a delay because I was using a VPN when I paid, so it got held for "fraud review". Told Arch to start moving data across. Updated my domain registrar to point to the new nameservers, but the change may take a while. Then realized I probably did that too soon. And my main email service hangs off those nameservers, since I use my domain for my email address. Went in and copied a few key DNS entries (those for email service) from the old account. Should be okay if my email is down for a while, both web-hosting services and my email-hosting service have alternate addresses for me, and I can log in to any of their sites to communicate with them too.

Within hours, someone says they're getting a "this site is not secure" message when accessing my web site. I don't see that. Soon another person says it too. The DNS in my VPN must be slower to update.

Within 18 hours, data has been transferred from old to new. Logged in and checked DNS records. All well, but soon I noticed the AAAA records point to IPv6 address on old server.

If I turn off VPN, ping goes to new hosting service. Checked that my web pages work. Did SSL tests. Checked that email works.

cPanel supports TOTP 2FA, but enabling it is in an odd place, in the Security section among various site features, not under the account profile pull-down menu. Main account supports TOTP 2FA also.

Filed a Support ticket about DNS records and SSH. Quick response: they do support IPv6 and will fix the records. And soon they gave me the information to get SFTP working. A couple of hours later, VPN's DNS has picked up my site's new IPv4 address, but the new IPv6 address still is working its way through the systems.

To-do: why their email host in my records, DKIM on their email, do they host as onion also ? Some image files give 403 error, maybe because I uploaded them just a few hours ago ?

According to cPanel, on my account there is a limit of 300K files, but no limits on disk usage or bandwidth.

Had Support remove a bunch of unneeded DNS records, and a couple of erroneous records copied from the old service.

As of 12/2019: "Our DNS cluster doesn't support DNSSEC unfortunately."
End of 1/2020: Wanted to change .htaccess file to get rid of "unsafe-inline" and use script hashes instead. But it turns out that lots of Google code blows up if you remove 'unsafe-inline' from 'style-src'. And if you have 'unsafe-inline' in a directive, you can't also have hashes. Changed from:

Header set Content-Security-Policy "script-src 'self' 'unsafe-inline' lots of Google domains;"
Header set Content-Security-Policy "style-src 'self' 'unsafe-inline' lots of Google domains;"

# script hashes:
#   contents of myscripts.js == 'sha256-something1'
Header set Content-Security-Policy "script-src 'self' 'sha256-something1' lots of Google domains;"
Header set Content-Security-Policy "style-src 'self' 'unsafe-inline' lots of Google domains;"
Use Report URI's "Script And Style Hasher" to generate hash values for your script code. Add those hash values to .htaccess file. Document code and corresponding hash in a comment in .htaccess file.

Have to change .htaccess file and upload to site, then go to site in Chrome, open a page, open DevTools (ctrl+shift+I), then do something to execute a script on the page. It will fail, and an error message will show the hash of the script. Add that hash to the .htaccess file, upload again, repeat.

Change .htaccess file and upload to site, then go to site in Firefox, open a page, Web Console is crtl-shift-K. See error messages.

Eventually figured out that I've been doing "Header set Content-Security-Policy" wrong all along: there can only be one such line in the .htaccess file, and all "-src" directives have to go in that one line. And then found out that using Google Ads just blows any restrictions wide open. Can't even restrict to a reasonable set of Google domains, because adservice.google.* varies by country of the client, and you can't have a domain pattern that ends with a *. Ended up having to set:

Header set Content-Security-Policy "default-src 'unsafe-inline' 'unsafe-eval' *;"

To get backup copy of DNS records, do CPanel / JetBackup / DNS Zone Backups.

To view numbers of unique visitors and page-views etc, do CPanel / Awstats, then:
Navigation / Viewed / Full list
Who / Locales / Full list
Referrers / Origin
Referrers / Search

Created my wife's web site on Wix

reddit's /r/Wix (beware scammers offering "help" and trying to phish your credentials)
Wix Support

Has a GUI site-designer (web app) that is okay. Provides lots of sample sites to get you started quickly. Pretty easy to figure out without reading docs. Easily supports multiple languages and mobile. Works okay with Firefox.

You can get started for free, and you can publish your site on Wix.com for free, but they will show small ads at the top. If you want to get rid of the ads, or use your own custom domain, cheapest plan is €4.50/month (€65/year with VAT).

They don't provide any way to export your design or HTML/CSS/JS, but they do provide two kinds of backups/snapshots on their site, so recovering from major site-editing mistakes should be easy.

Wix has limitations and quirks. For example, it seems they use absolute positioning, so if you're supporting multiple languages, and texts are different lengths in different languages, there will be white-space gaps in some pages in some languages.

People online complain about bad customer support from Wix, hard to get questions answered or problems resolved. But a quick look at the reddit sub seems to show Support occasionally being helpful there.

Beware: lots of scammers hanging around the Wix ecosystem. People offering to "help", either to phish your credentials or to get you into a paid-consultant situation.

Got the web site live. It's using HTTP, not HTTPS (found the place to turn it on, site dashboard / Overview / HTTPS, and it says HTTPS enabling is in process, automatically, wait an hour or two; it was using HTTPS later). The multi-language pull-down menu does not appear, in either Firefox or Chromium, and site is in English regardless of computer's default system language (found that the slide-settings in the language dashboard are misleading; I had them in "hidden" setting). Also there's still a Wix ad across the top, even though we paid for basic hosting, I thought the ad was supposed to go away (I remembered wrong; basic plan does have ads).

So, a few hours after paying for basic plan, connecting to our custom domain, and publishing, everything is working.

If you close the page and open it again, the language setting is not remembered, so I think they're not setting a cookie.

If you want to make a text block into a link (to a web page or email address or phone number), the text has to be a single line.

Images look incredibly blurred in Firefox 68.3.0 on Android 6, and the hamburger menu doesn't work. Tried changing Settings / Image Sharpening to "none"; didn't help. Tried changing Settings / Mobile Friendly to "off", and that made the images slightly better, but left the site with horizontal (desktop) pages on a vertical (mobile) screen. But then went to Chrome on another Android phone and the site is fine. Went back to my phone, used Chrome, site is fine. So they don't support Firefox on mobile properly, or Firefox has a problem.

For serious web-app-with-database-etc development, Wix has Corvid.

Alternative: Duda ? But it looks a little more complex than Wix, and 3x the price.

Alternative: Google Sites ? But it seems quite a bit more limited than Wix.


Interesting libraries:
adrai / flowchart.js (display flowchart)
Eugene_P's "My Top 13 JavaScript Diagram Libraries"
Hamza Ed-douibi's "10+ JavaScript libraries to draw your own diagrams"
Network diagrams:
xkln's "Creating Network Diagrams with JavaScript" (about Cisco's neXt UI Toolkit)

I tried Cisco's neXt: diagram always appears at end of page, docs seem broken, code is compressed, I think you're limited to one diagram per page ?, last update 4/2016.

Prism (syntax-highlight code blocks)

Online tool to change background color of a PNG to transparent: LunaPic

Kendra D. Mitchell's "How to Make GIF Background Transparent and Change Color" (OIE worked for me)

User click copies text to clipboard (site or app must have "clipboard-write" permission ?):

<span onclick="navigator.clipboard.writeText('This text will be copied')">Click here to copy</span>

User click copies text from clipboard (site or app must have "clipboard-read" permission ?):

<span onclick="navigator.clipboard.readText()
  .then(clipText => document.querySelector('.cliptext').innerText = clipText);
  ">Get contents of clipboard</span><br />
<span class="cliptext">contents of clipboard</span><br />

Time-wasting files:

You could put attractive nonsense files on your site to waste the time of bots or attackers.

Some people say this will be ineffective; bots are looking for very-low-hanging fruit such as vulnerable Wordpress plug-ins and plaintext credentials and open RDP ports.

Check with your hosting service to make sure this doesn't violate their TOS. My hosting service said: I can't host anything malicious, and a zip-bomb file qualifies as "malicious", not allowed.

Dr. Neal Krawetz's "It's All Fun and Games"

David Fifield's "A better zip bomb"
Make sure this won't blow up anti-virus on your site host.

"Give a 2 MB dump from /dev/rand that is AES-encrypted with a long random password."
AES-256 password is limited to 32 chars.
Secret Messages' "Encrypt Files"
You get a ZIP file with an AES file inside it.
UNZIP it to get the .aes file, set basename to something enticing, put on your site.

Give a password manager database that has a good master password and only junk contents. Important that it has a well-known file extension, so scanners can find it. Or (a lot of work) have the contents look like real account entries, but be bogus.

Give fake AWS credentials files:

sudo apt install awscli
aws --version

aws configure
# type AWS Secret Access Key ID such as AAAABBBBCCCCDDDDEEEE
# type default region name such as us-west-2
# choose "default output format"
cat ~/.aws/config
cat ~/.aws/credentials

aws ec2 create-security-group --group-name "n" --description "d"
# should get authentication failure

# copy config and credentials files to /.aws on your web site

Give fake Kubernetes Secrets file:

echo -n 'admin' | base64      # USERNAME
echo -n 'PASSWORD' | base64   # PASSWORD
Then create a file "secret.yaml" containing:

apiVersion: v1
kind: Secret
  name: mysecret
type: Opaque
  username: USERNAME
  password: PASSWORD
And put that file on the web site.

Give a ZIP file with a name such as "mypasswords.zip" that has a good master password and only junk contents:

zip -e mypasswords1.zip FILENAMES
zip -e mypasswords.zip mypasswords1.zip
rm mypasswords1.zip

VeraCrypt containers have no fixed header or standard file extension, so don't give a fake VC container, the attacker won't know what it is.

But: LUKS containers DO have a fixed header format. So maybe create an empty container file and put it on the site. See LUKS encryption section of my Linux Storage page.

Fake Linux /etc/shadow file: copy your real /etc/shadow file from your home PC, then edit to copy your normal-user encrypted password field to root's entry, then change same 10-20 chars in the middle of each encrypted password to random stuff.

Fake Linux /etc/passwd file: copy your real home PC /etc/passwd file, then make sure no personal info is in it. No need to edit it further; it has no encrypted passwords, and doesn't apply to the site host server anyway.

Fake Linux "pass" files:

sudo apt install gnupg2
sudo apt install pass

# https://fedoraproject.org/wiki/Creating_GPG_Keys
# https://www.fossmint.com/pass-commandline-password-manager-for-linux/

# creates ~/.password-store

pass generate FBUSERNAME/facebook.com 30
# failed on my system; something broken in gpg
# maybe because I uninstalled seahorse ?

pass insert FBUSERNAME/facebook.com
# and type a password, twice
# failed on my system; something broken in gpg
# maybe because I uninstalled seahorse ?

pass ls
Then copy the whole .password-store tree to your web site.

Crypto wallet. Install Electrum app, get passphrase and set password, File / Save Copy of wallet, get it from ~/snap/electrum/2/.electrum/wallets/ directory, also do Wallet / Private Keys / Export, get /snap/electrum/2/electrum-private-keys.csv, put them in /.bitcoin directory on web server.

Make sure the files are accessible to public. Also a good idea to allow directory listing to public.

Maybe create a password-protected directory, if the hosting service supports this:
The following is not best practice, but may be the best you can do on a shared hosting service.

For Apache server: Create a directory (with attractive name) you want to protect, and in that directory create a .htaccess file containing:

AuthType Basic
AuthName "Restricted Content"   # this will be displayed to user before login
AuthUserFile /.htpasswd-DIRNAME
Require valid-user

Use CLI command "htpasswd" (or 4WebHelp or Alterlinks ) to create a username:encyptedpassword string, put it into file /.htpasswd-DIRNAME, then set permissions to 600 or 640. (Best practice would be to put that file outside the site tree, but you can't do that on a shared host. Also you could name it ".htRANDOMCHARS" if you didn't want scanners to see it's a password file.)

Go to a browser and try to access "YOURSITE/DIRNAME". You should get prompted for username and password.

I wasn't able to get it working fully; I think the AuthUserFile path is wrong. But I left the site with the .htaccess file present and no .htpasswd file anywhere, and that works fine for my purpose: password-protected directory that's impossible to log into.

Make your site look like it's running Wordpress when it's not:
Create directories wp-content/plugins, wp-content/themes, wp-includes/js, wp-admin, wp-uploads.

Create files wp-includes/wlwmanifest.xml (maybe copy https://wpsmackdown.com/wp-includes/wlwmanifest.xml), license.txt (maybe copy https://wpsmackdown.com/license.txt), wp-links-opml.php (maybe copy and edit output of https://wpsmackdown.com/wp-links-opml.php), xmlrpc.php, wp-admin/wp-login.php, wp-admin/install.php, wp-config-sample.php, wp-config.php, readme.html

Tricky: Since my web server doesn't actual execute PHP, there is no harm in putting PHP files on my site. And the wp-links-opml.php file I put there is actually the output of the execution of a typical wp-links-opml.php file. Same for xmlrpc.php file.

Add to pages:

<meta name="generator" content="WordPress 3.0" />
<!-- This site is optimized with the Yoast SEO plugin v12.5 - https://yoast.com/wordpress/plugins/seo/ -->
<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="https://s0.wp.com/wp-includes/wlwmanifest.xml" />

<img src="https://s0.wp.com/wp-content/themes/vip/99u-2016/assets/img/logo-white-adobe-hover.svg" alt="adobe" width="0" height="0" style="visibility:hidden" />

Install themes: Go to https://wordpress.org/themes/ , select a theme, click Download button on it, download ZIP file, save ZIP file into wp-content/themes. Extract contents, producing directory THEMENAME and files inside it. Delete ZIP file. Go into THEMENAME directory and delete all files except index.php and style.css . On your site, create directory wp-content/themes/THEMENAME and upload index.php and style.css to it.

To get a known-vulnerable theme, I took 1.2.2 from https://github.com/themegrill/flash/releases

Use https://whatcms.org/ and https://cmsdetect.com/ and https://scanwp.net/ and https://www.wpthemedetector.com/ and https://whatwpthemeisthat.com/ to test the site. https://www.billdietrich.me/index.html curl --data "param1=value1&param2=value2" https://wpsmackdown.com/xmlrpc.php

Thorsten Frommen's "How to Detect WordPress Websites"