Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Site generation slows down for large sites #144

Closed
qmfrederik opened this issue Oct 30, 2019 · 10 comments
Closed

Site generation slows down for large sites #144

qmfrederik opened this issue Oct 30, 2019 · 10 comments
Assignees

Comments

@qmfrederik
Copy link

I'm trying to use Hugo + Docsy to generate a site for a library. The idea is that the site includes a reference section, with individual pages for classes, methods and so forth.

I've found that the site generate slows down drastically as you add more pages to the site. For example, generating a site with about 3100 pages takes close to one minute.

That seems a bit excessive to me, so I wanted to check back here and see if that's expected behavior?

Here's the output of hugo --templateMetrics:

Building sites … 
  Replace Autoprefixer browsers option to Browserslist config.
  Use browserslist key in package.json or .browserslistrc file.

  Using browsers option cause some error. Browserslist config 
  can be used for Babel, Autoprefixer, postcss-normalize and other tools.

  If you really need to use option, rename it to overrideBrowserslist.

  Learn more at:
  https://github.com/browserslist/browserslist#readme
  https://twitter.com/browserslist



Template Metrics:

     cumulative       average       maximum         
       duration      duration      duration  count  template
     ----------      --------      --------  -----  --------
  4m14.652164671s   92.399188ms  567.255586ms   2756  partials/sidebar.html
  4m9.198382732s  161.607252ms  566.054856ms   1542  partials/sidebar-tree.html
  3m36.378580825s    89.56067ms  1.276525793s   2416  docs/single.html
  1m12.36804656s  212.847195ms  1.372857912s    340  docs/list.html
  12.330060444s   36.264883ms  146.511833ms    340  partials/section-index.html
  10.824119209s    3.920361ms  1.022153757s   2761  partials/head.html
   8.158249879s  1.019781234s  1.020271602s      8  partials/head-css.html
   1.572953639s     569.704µs   17.125308ms   2761  partials/navbar.html
   983.081689ms     356.705µs   16.259392ms   2756  partials/breadcrumb.html
   598.739322ms      217.17µs   36.957801ms   2757  partials/scripts.html
   579.925159ms     210.041µs   10.358824ms   2761  partials/footer.html
   382.747408ms     138.877µs   16.937873ms   2756  partials/toc.html
   378.603617ms     156.706µs    6.807401ms   2416  _default/content.html
   368.573735ms    1.071435ms   15.249291ms    344  _default/list.rss.xml
    179.93975ms       65.29µs       6.466ms   2756  partials/page-meta-lastmod.html
    159.61306ms      57.914µs   15.059782ms   2756  partials/page-meta-links.html
    92.980861ms      21.608µs    1.624397ms   4303  partials/search-input.html
    54.420419ms   54.420419ms   54.420419ms      1  _internal/_default/sitemap.xml
    41.586507ms      15.062µs    3.813495ms   2761  partials/hooks/head-end.html
    34.034045ms      12.344µs    1.741097ms   2757  partials/hooks/body-end.html
    11.737949ms    3.912649ms    11.66105ms      3  _internal/shortcodes/relref.html
     6.935675ms    2.311891ms    4.534691ms      3  _default/list.html
      4.55139ms      42.142µs     521.871µs    108  shortcodes/alert.html
     2.272225ms      45.444µs     917.529µs     50  _internal/shortcodes/param.html
      1.70778ms      853.89µs    1.377925ms      2  shortcodes/blocks/cover.html
     1.638304ms      68.262µs     799.477µs     24  shortcodes/download.html
     1.550492ms      46.984µs     774.512µs     33  shortcodes/pageinfo.html
     1.523481ms    1.523481ms    1.523481ms      1  home.html
     1.306066ms    1.306066ms    1.306066ms      1  404.html
     1.080756ms     180.126µs     794.333µs      6  shortcodes/limd_download.html
      1.05589ms     263.972µs     946.317µs      4  shortcodes/blocks/section.html
      838.995µs     838.995µs     838.995µs      1  shortcodes/blocks/lead.html
      783.637µs      52.242µs     702.001µs     15  shortcodes/notice.html
      730.356µs     365.178µs     725.524µs      2  shortcodes/attachments.html
       191.82µs      191.82µs      191.82µs      1  partials/favicons.html


                   |  EN   
+------------------+------+
  Pages            | 3105  
  Paginator pages  |    0  
  Non-page files   |    4  
  Static files     |   36  
  Processed images |    4  
  Aliases          |   80  
  Sitemaps         |    1  
  Cleaned          |    0  

Total in 39463 ms

@LisaFC
Copy link
Collaborator

LisaFC commented Nov 8, 2019

Huh, that's strange, one of the reasons I use Hugo is because of how fast it is - let me investigate.

@LisaFC LisaFC self-assigned this Nov 8, 2019
@LisaFC
Copy link
Collaborator

LisaFC commented Nov 11, 2019

Hi there, can I check a few things? What are you running hugo on (platform, OS, etc.) and can I have a look at the repo?

@qmfrederik
Copy link
Author

@LisaFC Sure thing!

Here's an example repo: https://github.com/qmfrederik/docsy-perf . It's essentially the sample site, with a new commit which adds auto-generated documentation.

This is on Ubuntu 18.04 (x64), with a recent version of Hugo:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
$ hugo version
Hugo Static Site Generator v0.59.0-1DD0C69C/extended linux/amd64 BuildDate: 2019-10-21T09:45:38Z

@LisaFC
Copy link
Collaborator

LisaFC commented Nov 14, 2019

Ok, I tried on Debian with Hugo 0.55 extended and I'm getting almost identical timings. It looks like it's the time it's taking to build the autogenerated left nav with that many nested pages, a huge amount of time seems to be spent in the sidebar. However, that partial should be cached. I'll poke around with it a bit more.

@LisaFC
Copy link
Collaborator

LisaFC commented Nov 14, 2019

So taking the partialCached out made it even slower...

It's definitely building the sidebar though because if I took that out your site built in 4 seconds. Let's see if it's possible to speed things up a bit.

@Resonanz
Copy link

Gripping developments, but what happened next? :-)

@LisaFC
Copy link
Collaborator

LisaFC commented Jan 6, 2020

What happened next is that I still haven't managed to make it any faster. :( I suspect just building a big nested navbar on the fly with Javascript is just kind of time-consuming. Will give it another go when I have a moment, or happy for someone else to try!

@tshort
Copy link

tshort commented Mar 10, 2020

I'm also starting to hit this. Has anyone found a different theme where sidebar generation is faster? If so, maybe we can review their code.

@tshort
Copy link

tshort commented Mar 10, 2020

I changed the section-tree-nav-section template in partials/sidebar-tree.html as follows. This reduces the amount of recursion when generating the sidebar. The code in master is including entries for cousin pages in the sidebar, but it was hiding them, so there's no point in including them. The {{ if $show }} conditional is the key change. On a site I'm working on, that reduced the build time from 30 secs to 11 secs. It especially helps with sidebar_menu_compact = true.

{{ define "section-tree-nav-section" }}
{{ $s := .section }}
{{ $p := .page }}

{{ $shouldDelayActive := .delayActive }}
{{ $active := eq $p.CurrentSection $s }}
{{ $show := or (and (not $p.Site.Params.ui.sidebar_menu_compact) ($p.IsAncestor $s)) ($p.IsDescendant $s) }}
{{ $sid := $s.RelPermalink | anchorize }}
<ul class="td-sidebar-nav__section pr-md-3">
  <li class="td-sidebar-nav__section-title">
    <a  href="{{ $s.RelPermalink }}" class="align-left pl-0 pr-2{{ if not $show }} collapsed{{ end }}{{ if $active}} active{{ end }} td-sidebar-link td-sidebar-link__section">{{ $s.LinkTitle }}</a>
  </li>
{{ if $show }}
  <ul>
    <li class="collapse show" id="{{ $sid }}">
      {{ $pages := where (union $s.Pages $s.Sections).ByWeight ".Params.toc_hide" "!=" true }}

      {{ $pages := $pages | first 50 }}
      {{ range $pages }}
      {{ if .IsPage }}
      {{ $mid := printf "m-%s" (.RelPermalink | anchorize) }}
      {{ $active := eq . $p }}
      <a class="td-sidebar-link td-sidebar-link__page {{ if and (not $shouldDelayActive) $active }} active{{ end }}" id="{{ $mid }}" href="{{ .RelPermalink }}">{{ .LinkTitle }}</a>
      {{ else }}
      {{ template "section-tree-nav-section" (dict "page" $p "section" .) }}
      {{ end }}
      {{ end }}
    </li>
  </ul>
{{ end }}
</ul>
{{ end }}

@emckean
Copy link
Collaborator

emckean commented Jun 2, 2023

Closing this as no activity for 2 years -- please feel free to reopen if there's more to report!

@emckean emckean closed this as completed Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants