Zenspider/warnings double loads and fixes #1962
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(no good way to set $VERBOSE from the outside)
Drops `rake check:specs VERBOSE=1` from 331 lines to 144.
load will load, always. Don't use it unless you really really mean it.
Most of these should just be removed instead of underscored... but I want eyeballs on this first. Some aren't being used the way they think they're being used.
Turns out it was one local variable used in a lot of regexps. This updates the link to the reference and fixes it per reference.
git gem has warnings, this quells them
…quire Bundler shouldn't require everything. Slows down the world.
|
One warning is left for a regexp in julia that I am digging into but might best me: I've figured out that it is it is the last half, and related to the |
|
Ya know, I've been against it in the past, but I wouldn't mind transitioning to |
|
That being said, I would not trust ruby's |
jneen
requested changes
May 17, 2023
| Kernel::load File.join(Lexers::BASE_DIR, 'viml/keywords.rb') | ||
| self.keywords | ||
| end | ||
| require_relative "viml/keywords" |
There was a problem hiding this comment.
Hm. The reason we did it this way was that these giant keyword files have a massive memory footprint, and we don't want Rouge to load them unless the lexer is actually in use. I think it'll still work if we use require_relative dynamically maybe?
There was a problem hiding this comment.
running the above benchmark code with /usr/bin/time -lph:
master:
61194240 maximum resident set size
3888 page reclaims
4 page faults
17 involuntary context switches
6141564240 instructions retired
1349472140 cycles elapsed
55493952 peak memory footprint
mine:
59604992 maximum resident set size
3791 page reclaims
4 page faults
10 involuntary context switches
6451778027 instructions retired
1398372284 cycles elapsed
51660096 peak memory footprint
(removed lines reporting 0)
I reran each 3 times and the numbers are all consistent...
| @@ -115,7 +115,7 @@ def self.keywords_type | |||
| rule %r/"/, Name::Variable, :double_string | |||
| rule %r/`/, Name::Variable, :backtick | |||
|
|
|||
| rule %r/\w[\w\d]*/ do |m| | |||
| rule %r/\w+/ do |m| | |||
There was a problem hiding this comment.
This is probably a mistake and they meant either \p{L}\w+ or [a-zA-Z]\w+, would have to check the language spec for that.
| @@ -73,7 +73,7 @@ def initialize(opts={}) | |||
| end | |||
|
|
|||
| state :export do | |||
| rule %r/[\w[\$]{1,2}{}()-]/, Name::Variable | |||
| rule %r/[\w\${}()-]/, Name::Variable | |||
There was a problem hiding this comment.
Not really this PR's fault, but we usually ask for + at the end of single-char-class regexes like this for perf reasons. This one's old enough it might have been my fault 😅
There was a problem hiding this comment.
I'm fine adding the + ... but:
do you see how the {1,2} is also inside of a class, so it isn't a length specifier at all?... you might want to change the regexp to match more correctly. I haven't studied the make grammar in a really long time, but I'm pretty sure the above (either version) is incorrect.
| Kernel::load File.join(Lexers::BASE_DIR, "llvm/keywords.rb") | ||
| types | ||
| end | ||
| require_relative "llvm/keywords" |
There was a problem hiding this comment.
Similar to before, we'd like to avoid loading these large keywords files for users who aren't highlighting LLVM. Though now I look at this it appears to have been a copy paste job and it will absolutely load that file multiple times 🤦
There was a problem hiding this comment.
The old code is wrong. It shouldn't use load. If that alone gets fixed, great... but seriously, The strategies you used 10 years ago to speed things up aren't the strategies you need today. Case in point:
$-w = nil
$: << "lib"
require "rouge"
keywords = Dir["lib/rouge/lexers/*/keywords.rb"]
iters = 30
t0 = Time.now
iters.times do
keywords.each { |f| Kernel.load f }
end
t1 = Time.now
puts "Average time to load all keywords.rb: %8.5fs" % [(t1 - t0) / iters]
# Average time to load all keywords.rb: 0.00812sIt costs nearly nothing to load those keyword files... granted, every machine is different, but still. I suggest you measure and reevaluate whether self modifying code is worth it. I had thousands of lines of warnings when I was first evaluating whether to switch to rouge or not. Literally thousands. Those warnings have a cost too.
There was a problem hiding this comment.
If you really want to make rouge faster, don't load everything. Load (roughly) nothing. Push it towards a lazy loading system. But if you're going to load everything, let ruby do it and let it do it properly (and only once)
Nobody is claiming this... Lack of output from |
|
Where does this stand? |
|
2 more weeks after months... Do what you want with this PR. I'm divesting myself of it. |
|
Okay! |
|
Thank you for your hard work on this @zenspider 🙇🏼 ❤️ Sorry I am just catching up to this. I think the |
This was referenced Jan 7, 2024
This was referenced Mar 17, 2024
|
Hello. 👋 I'm running into this problem when warnings are enabled in my test environment. Here's what I'm using: Implementation# frozen_string_literal: true
require "asciidoctor"
module Milestoner
module Renderers
# Renders ASCII Doc as HTML.
class Asciidoc
SETTINGS = {
safe: :safe,
attributes: {
"source-highlighter" => "rouge",
"rouge-linenums-mode" => "inline"
}
}.freeze
def initialize settings: SETTINGS, client: Asciidoctor
@settings = settings
@client = client
end
def call(content) = client.convert content, settings
private
attr_reader :settings, :client
end
end
endSpecrequire "spec_helper"
RSpec.describe Milestoner::Renderers::Asciidoc do
subject(:renderer) { described_class.new }
describe "#call" do
it "answers HTML code block" do
html = renderer.call <<~CONTENT
[source,ruby]
----
1 + 1 = 2
----
CONTENT
expect(html).to eq(<<~PROOF.strip)
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><span class="mi">1</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">=</span> <span class="mi">2</span></code></pre>
</div>
</div>
PROOF
end
end
endWarningsThis is what I see in my console: If I remove these attribues in my implementation: attributes: {
"source-highlighter" => "rouge",
"rouge-linenums-mode" => "inline"
}...the warnings go away. I don't think this is an Asciidoctor problem but with Rouge itself. It looks like this work is partially complete but I would expect these warnings to show up in the Rouge test suite when warnings are enabled? I can make these warnings disappear by using the Warning gem to silence them for only the Rouge gem but it'd be great if this is fixed in Rouge proper. Anyway, hope this helps. |
|
Hello everyone, I know I am late to the party (I may have missed the notification from this an year ago), but to anyone coming across this pull request, the reason why the codebase contains numerous The ideal solution here would be to have a custom loader that calls Or, just delegate the loading process to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1961
This PR fixes >300 warnings that are emitted while loading the code and/or running specs with warnings on.
Please consider reviewing this by commit.
Also consider turning off whitespace changes (for one commit on a rake task file). Tack on
?ws=1.This is 12 commits, some should be squashed but I separated them because I didn't trust all my changes and this lets it be reviewable by the language owner.
Note: This completely removes use of (but not the definition of)
Rouge.load_fileand friends. They were not properly preventing double loads and eventually I decided it would be cleaner and easier if everything was just required via ruby to be more idiomatic and quell ~200 redefined method warnings. This does remove all of the "self-mutating" methods and eagerly loads the keyword files. It does not noticeably change load time at all. That might have been relevant when we all had slow spinning hard drives, but not anymore.