(del / no need to keep, only confusing for future readers)



Edited 1 time(s). Last edit at 08/07/2019 01:04PM by siria.

Quote
siria
Hm, you're explaining me the old code, the part that I understand.
But what means \"([^]+)\" in your new code? Is it perhaps something like the allpurpose \"(.*?)\"

Oh, my...

A mistake, just a mistake. Anyway, that won't take any effect.

Fixed.

They really changed this time.

No hope for a fix.

I'm trying to reverse engineering it, but as I don't have a sample of how it was before today, I have no clue what actually the script does. Who did it really complicated things a lot. How easy could have been using eval to parse JSON, or JSON.parse...

Lets see if someone around the net makes a parser to be used :/ Since then, lets go back to use the download web pages



Edited 2 time(s). Last edit at 09/11/2019 08:04PM by JohnHell.

Hey JohnHell,
Rarely visiting that site, but tested your macro in 2018 and 2019 with new and old KM/Gecko to find engine bugs. Stripped some garbage, but nothing essential. Download still works with 2018 and 2019 macro versions today. UA is important.

Quote
anonymous
Hey JohnHell,
Rarely visiting that site, but tested your macro in 2018 and 2019 with new and old KM/Gecko to find engine bugs. Stripped some garbage, but nothing essential. Download still works with 2018 and 2019 macro versions today. UA is important.

Looks like they have restored the old code again. Believe me, it wasn't the UA. The UA has been the same today from starting to end.

In fact, this other script I found changed the same exact portion of code to catch what YouTube changed. I bet tomorrow they change it again. Check the history for 0.9.7:
greasyfork.org/en/scripts/369400-local-youtube-downloader/versions

I don't get what do you mean by testing the macro to find engine bugs

The Script is not mine and the macro is just injecting JS. Too simple. Not a big deal to test with ;P There are other bigger/harder/complicated macros out there to test K-meleon against to find bugs XDD

Indeed that script had garbage. A lot, from Greasemonkey and other trash.



Edited 1 time(s). Last edit at 09/12/2019 01:05AM by JohnHell.

Great. Save the page next time it happens.

What for? As a proof? xDDD

After all I'm not the maintainer of that script. Only little fixes if I can. When they change the whole code, I pass.

It is better to find another script or a website than spend a lot of time in reverse engineering it. Effort doesn't worth the unlikely benefit.

Forget the script. A snapshot of that flaky page may reveal their plans for the future.

Russian comrades created the ViewTube stripped down version that can be injected into KM

Thanks, but stubborn me, until I'd try myself when I had peace of mind, I moved to bitdownloader.com, that siria suggested time ago, and now here it is a new change to the script.

First page:
http://kmeleonbrowser.org/forum/read.php?9,142050,page=1#msg-142252

It took me... quite a bit with all reverse engineering, YouTube new code and the trash code the script has itself.

Ok, the code maybe doesn't have trash code. The author had it reasons, and one was make it compatible with every browser (kudos), but figure, I removed almost 4KB by simplifying the main parsing waaaaay a lot. I don't know why he didn't do that way the first time

From that removal code (including trash code not used by injectJS and only by Greasemonkey) I removed the first commentary of the file and Greasemonkey settings. They are not needed for us. Now the file is even below 32KB.

That makes the code not compatible with other browsers, but, hey, this is intended for K-meleon, not others.

I hope it works for you, siria, on 1.6. Should work, but don't forget to add the line to display the download options as you did in the first page. In a quick test I couldn't manage to display it

Forget the above, siria, it works. I didn't remove the display none as I instructed you years ago LOL. Sorry, tired XD



Edited 5 time(s). Last edit at 01/25/2020 04:43AM by JohnHell.

Added a little change to prevent, or predict, changes on the code when JavaScript is disabled in K-meleon, because I bet they will change it, and with a fixed hardcoded number for an index will get future troubles.

Even though the changes by YouTube might have come by the changes in their policies about audience fine tuning (for kids (uploaders already know what is that about)) and I don't bet a new change soon. And if the changes weren't by this and just code cleaning, the timing has been absolutely perfect.

Thanks! Have noticed just a few days ago that it's broken again, and was hoping you could fix it again
Haven't tried yet, will need more time, and then adjust to my own old, long since forgotten tweaks again. For example am plastering all available download links just openly in the upper left page corner, and (not needed now anymore?) had split the script into 2 parts, then stitched together again by macro.

Just as general side note regarding readfile-by-macro, in the very last gecko-KM 76RC2 Dorian increased the limit from 32kB to 64kB (along with the limit for injectJS-return, from 4 to 32)



Edited 1 time(s). Last edit at 01/25/2020 08:40PM by siria.

I updated the file a little removing a typo I found using the FC windows command (file comparison); and with a non-dramatical change in the way, when JavaScript is disabled, it parses and gets the code.


It is a very low priority update. It is up to you. Nothing that the script won't work today or tomorrow.


I made those changes while I was trying to get the copyrighted videos to download (those with some copyrighted content, as with background songs, for example), but the signature decryption that expect the scripts is not there anymore, so I have no clue how to do it. I should have removed that code (a few KBs), but... I don't feel confident about it. Too many dependencies here and there.

Having a look to the script I posted back doesn't help to make it work when it uses the latest EcmaScript 6 functions and methods. It gets dizzy trying to figure it out. So, I leave it as is, despite being useless code. Better use savefrom to download those files if the download menu isn't displayed. For the rest, it works.

Hey John,
Downgraded UA and reviewed your update because of the deprecation warnings on the boards. Still some white space followed by line endings and this time that superfluous BOM didn't tell your text editor to use UTF-8.

Quote
anonymous
Downgraded UA and reviewed your update because of the deprecation warnings on the boards.

What deprecation warning on what boards are you talking about that made you downgrade the User Agent and review the update?

Sorry, didn't get what you are trying to say.

Quote
anonymous
Still some white space followed by line endings and this time that superfluous BOM didn't tell your text editor to use UTF-8.

Sometimes K-meleon refuses to load the scripts (specially in my language) if there is no BOM. Don't ask me why. And that happened this time, so I had to save it with BOM and I carried it to the new file, I guess. I don't know, to be fair.

For some editors, anyway, is not superfluous. They need it. The editor I use doesn't, but as it hasn't a method to change the binary encoding of the file, I use the easy way, notepad and... BOM (sorry for the joke). I could use Notepad++, but I don't get used to it, despite the popular it is; and doesn't help with what I said in the paragraph above with that requirement some times. I don't know why they don't load sometimes if I don't add a BOM.

Also, if they don't have BOM, characters aren't displayed correctly with the font fallback.


What is that white space you find?

I upload a new one whithin the following minutes without the BOM.



Edited 1 time(s). Last edit at 02/09/2020 08:21PM by JohnHell.

OK, updated file.

@JohnHell
The warning was in Siria's last post with a link to MSFN. White space like "\ " or "\t" is found at end of line 36 of your latest script.

Quote
anonymous
@JohnHell
The warning was in Siria's last post with a link to MSFN.

:-?

I'm getting lost again.

What post by siria with an MSFN link? There is no one in this thread Is this one (latest to date)?
http://kmeleonbrowser.org/forum/read.php?19,148500,151690#msg-151690

The thing is that if this is all about the script not compatible with the new layout. I'm going to be fair. I haven't even had look to the new layout and I didn't even consider to have a look until they force it in March (or so I've read somewhere).

There are "tricks", as the explained in the previous page to make the menu appear permanently, if anyone has troubles, until I modify it again, but I'm not in that ship yet. Probably the best solution then would be just that. Place it at the very top or footer and forget about hardcoded expected elements and even if the code hasn't changed again and working urls can be extracted.

Despite I look like the code maintainer, I just use it and share it with you, even this code+macro isn't the one I use myself.

Quote
anonymous
White space like "\ " or "\t" is found at end of line 36 of your latest script.

:-?

Line 36 in the current .js file is:

  var isDecodeRuleUpdated=false;

I didn't even smell that portion of code. Remember that the script is not mine. I'm just making adaptions. And checking back the versions I keep, those spaces at the end are from the very beginning of the code itself, as you can check here:
https://web.archive.org/web/20160121123600/https://raw.githubusercontent.com/gantt/downloadyoutube/master/script/yt.user.js

There isn't a problem with them, except for 2 bytes..., "wasted"? (the original coder wasted more bytes as he doesn't use tabs (a nightmare for me) but spaces for code formatting).

Yeah it's all about Google's killing the Classic View completely soon, and my link to MSFN has more info, and meanwhile with some follow-up posts (someone found that "modern" browsers need plugin-container now to see videos, and multiprocess e10s or whatever greatly speeds it up, but KM is too far behind anyway).

Just change your UA in your current main browser to e.g. FF99 and take a quick look.
The future code is not just some cosmetical changes, but it's fundamentally different, as also stated by Google themselves.

Quote
siria
Yeah it's all about Google's killing the Classic View completely soon, and my link to MSFN has more info, and meanwhile with some follow-up posts (someone found that "modern" browsers need plugin-container now to see videos, and multiprocess e10s or whatever greatly speeds it up, but KM is too far behind anyway).

That is for AV1, according to the thread, that you shouldn't care about. I doubt you could even play in Win98, even if you have enough hardware, because AV1 and x265 requires quite a powerful hardware.

Quote
siria
Just change your UA in your current main browser to e.g. FF99 and take a quick look.
The future code is not just some cosmetical changes, but it's fundamentally different, as also stated by Google themselves.

If the code with the video URLs isn't there, the layout is the less you should care about, because you'll need to go to the online downloader pages.



You are so insisting that I take a look, and, what?, all empty; 2.5MB of the page itself and a total of almost 8MB with external JS and CSS just to watch a single video.

God bless optimization

Now I understand why mobile data plans are sold as popcorns. What a waste of data transfers (yeah, is gziped, but still about 1.5M


Anyway, the code still looks there... for now, somewhere between those 2.5MB XDDDD.

As I said, I'm not going to make anything until they force. I'm not in a hurry. Explanations to place the menu at the bottom are already in the first page, if someone needs to use the new layout.





Edited 3 time(s). Last edit at 11/07/2020 02:55AM by JohnHell.

Quote
JohnHell
Anyway, the code still looks there... for now, somewhere between those 2.5MB XDDDD.
As I said, I'm not going to make anything until they force. I'm not in a hurry. Explanations to place the menu at the bottom are already in the first page, if someone needs to use the new layout.

Are you saying our old script can still extract the mp4-links from that mess??
If yes, that gives some hope now But I'd need that script to still work without page JS (only injectJS), or will only freeze...
And do you think it'll work with polymer v2 too? They say v1 will not last very long either, perhaps a year, and then only v2 or mobile will be available.
After my first quick check some days ago, I'll probably be able to extract and display basic text info, description, author, date etc., plus the static image, but dreading the horrible code work, just due to almost complete lack of JS skills. Am sure it can be done a LOT easier, but had to waste half a year to get instagram (similar json stuff) working as I like, completely convoluted by getting whole page innerHTML as a string and extracting stuff with regex, then puzzling together a whole new html source. Completely idiotic but no clue how to get that %$&!! json "string" as a still working "script" Still slightly traumatized of that endless struggle and neither enough time nor nerves for a similar nightmare for YT now. At least this time I could probably recycle some insta-macro code instead of starting completely at zero again, but still, the thought alone is horror. So yeah, probably will need 3rd party download sites, if (?) those keep working. Grmpf.

BTW there's a second topic on msfn, specifically for old browsers:
https://msfn.org/board/topic/181184-yt-may-not-work-on-old-browsers-anymore-starting-march-2020/#comments



Edited 2 time(s). Last edit at 02/10/2020 07:42PM by siria.

Quote
siria

Quote
JohnHell
Anyway, the code still looks there... for now, somewhere between those 2.5MB XDDDD.
As I said, I'm not going to make anything until they force. I'm not in a hurry. Explanations to place the menu at the bottom are already in the first page, if someone needs to use the new layout.

Are you saying our old script can still extract the mp4-links from that mess??

They are in the JSON object and the JSON text is still there... still.......

Quote
siria
If yes, that gives some hope now But I'd need that script to still work without page JS (only injectJS), or will only freeze...

Always has worked this way. The JS is needed only to display the menu. I won't enable JS myself for that bunch of megabytes of JS API called polymer/material. No, thank you. So I should include a NO JS by default for all, as I have it running for myself.

Quote
siria
And do you think it'll work with polymer v2 too? They say v1 will not last very long either, perhaps a year, and then only v2 or mobile will be available.

You only need a place to put the download menu. It can be just the body and position it elsewhere.

Quote
siria
After my first quick check some days ago, I'll probably be able to extract and display basic text info, description, author, date etc., plus the static image, but dreading the horrible code work, just due to almost complete lack of JS skills. Am sure it can be done a LOT easier, but had to waste half a year to get instagram (similar json stuff) working as I like, completely convoluted by getting whole page innerHTML as a string and extracting stuff with regex, then puzzling together a whole new html source. Completely idiotic but no clue how to get that %$&!! json "string" as a still working "script" Still slightly traumatized of that nonsense and neither enough time nor nerves for a similar nightmare for yt now. So yeah, probably will need 3rd party download sites, if (?) those keep working. Grmpf.

JSON.parse is what you need. Then is just an object. Before I understood it, it was complicated to me too. Remember back when Dorian asked to use it in the skinning system. It is the same structure.
https://www.w3schools.com/jsref/jsref_obj_json.asp

Available back to Firefox 3.5, so K-meleon 1.6:
https://www.w3schools.com/jsref/jsref_parse_json.asp

Format it with this, for easy reading:
http://jsonviewer.stack.hu/

All the data is in JSON format in the page, or almost. But, really, don't waste your time in converting everything. Or do whatever you want, but will be a pure nightmare.

I enjoyed in the past the dynamic elements on pages (back 20 years), but this nonsense of all page in JavaScript via JSON is crazy.

The good side is that you could parse and rebuild the page without JavaScript with JS injection, but it is too much as you discovered with Instagram. Depending if you want to extract all, of course.

Quote
siria
BTW there's a second topic on msfn, specifically for old browsers:
https://msfn.org/board/topic/181184-yt-may-not-work-on-old-browsers-anymore-starting-march-2020/#comments

MMMhhh, that's the same you posted earlier.



Edited 1 time(s). Last edit at 02/10/2020 07:59PM by JohnHell.

> JSON.parse is what you need. Then is just an object.

Yeah in theory, but as mentioned 1-2 times already, in practice my probs with json already start with the very first step: how to get that **** into a var???

var i, HitX, LIST=document.getElementsByTagName('script');
var Hits=LIST.length;
for (i=0; i < Hits; i++) {
HitX=LIST[i];
alert(i+' = '+HitX);
if (HitX.ytInitialPlayerResponse) alert('ytInitialPlayerResponse FOUND in: '+i);
}
alert('scripts: '+Hits);



Edited 1 time(s). Last edit at 02/11/2020 03:01PM by siria.

Quote
siria
> JSON.parse is what you need. Then is just an object.

Yeah in theory, but as mentioned 1-2 times already, in practice my probs with json already start with the very first step: how to get that **** into a var???

var i, HitX, LIST=document.getElementsByTagName('script');
var Hits=LIST.length;
for (i=0; i < Hits; i++) {
HitX=LIST;
alert(i+' = '+HitX);
if (HitX.ytInitialPlayerResponse) alert('ytInitialPlayerResponse FOUND in: '+i);
}
alert('scripts: '+Hits);

You need to get what is between curly brackets {}, and the curly brackets too.

If a script is "var x = {xxxxxx};" you need to take "{xxxxxx}" by JSON.parse to a variable like "var myjson = JSON.parse('{xxxxxx}');".

You can use HitX.search(/regex/) to find the exact beginning and end of the text to extract with String.substring(index1,index2); (string is HitX text/string).

Then to extract data: "myjson.somedata" and if somedata holds a new object with a property called newdata "myjson.somedata.newdata" and if it is an array "myjson.somedata[indexofthearray]". And so on.



Edited 3 time(s). Last edit at 02/11/2020 03:59PM by JohnHell.

Hmm, highly complicated but now dawning a little... And also meanwhile found this in the userscript:
args = JSON.parse(JSON.parse(bodyContent.substring....
Sigh, kinda steep mountain to climb, but now got the first lesson: that json-string is really just a pure string! Okay... the first mm forward ;-)

Quote
siria
Hmm, highly complicated but now dawning a little... And also meanwhile found this in the userscript:
args = JSON.parse(JSON.parse(bodyContent.substring....

That is a YouTube thing.

There are two JSON.parse because the JSON portion is needed is encoded, in the script itself, as a string. It is not an object ready to use. So you (as YouTube must do too) need to convert that string back to object again.

It is the same as when JavaScript is enabled (the true condition), but as when JavaScript enabled the object already exists..., it is used only once.

Quote
siria
Sigh, kinda steep mountain to climb, but now got the first lesson: that json-string is really just a pure string! Okay... the first mm forward ;-)

JSON is an object, so means between curly brackets, as the K-meleon skinning system, comma separated pair of values ({a:b,c:d,e:f}).

But YouTube, in the case above, or when capturing with injectJS, so JS disabled, the script is text (aka string), so you must convert it.

But, hey, you could do all in strings and just extracting by substrings... if you dare XD After all, this is what we do in K-meleon macros, as we don't have objects. The nearest, or equivalent, to objects are the inis, as they store pairs of values, but, instead a:b, a=b; and instead x.a, [sections]



Edited 2 time(s). Last edit at 02/11/2020 06:22PM by JohnHell.

> But, hey, you could do all in strings and just extracting by substrings... if you dare XD

No thanks, not anymore ;-) That's what my insta-macro does, but ouff.... Additional time-eating probs were that I also needed to figure out more html-stuff, and more css-stuff and etc, and of course have meanwhile everything forgotten again. If I'd known in advance exactly how high and hard that mountain was, considering my nonexistent skills, I'd never even started. And the icing on the cake was to discover, when almost finished, that there exist 3rd party services as webviewers!! Argh...

Another headache about that approach is the nesting prob. How to make sure to catch the matching closing bracket of several nested {xxx{xxx{xxx}xxx}xxx} brackets, and especialy how to make sure not to trip if someone tpyed such brackets as readable, plain text inside an xxx value.
But now have hope to finally get the values extracted 100% reliably using real json methods. When getting around...

Quote
siria
Another headache about that approach is the nesting prob. How to make sure to catch the matching closing bracket of several nested {xxx{xxx{xxx}xxx}xxx} brackets, and especialy how to make sure not to trip if someone tpyed such brackets as readable, plain text inside an xxx value.

First curly bracket { after an equal =

Last curly bracket } before semicolon ;

If you take wrong indexes, JSON.parse would tell you can't be parsed.

If curly bracket is inside a string, so between quotes, it is a string character, nothing else.

Okay, did some more reading, riddling and testing, and now yet more glad you could fix the script again, since I definitely wouldn't have managed that. Tried to at least understand your changes, but even this was far too complicated too. Well, gave up.

Next step: would really have liked to get "all" available video formats as download links in the Mobile Layout too, but no luck either. The script does nothing at all there.
Then tried to figure out how to get the included "debug" function to log errors, but no luck either: debug-function seems to only report anything at all if there are NO errors, very funny... and even this I can only verify via additional test alerts, since that stubborn debug-log-div keeps hiding! Where?? (and yeah I've changed the display style to 'block'). Tsss... never mind, gave this up too.

Next step: took a closer look at the included download links in the json strings. Finally a bit light at the end of the tunnel: those seem to be quite similar as in Classic View! And noticed the original userscript adds the title at the end of the URLs, but downloads seem to work just as well without adding those. Simply the URL as contained in the json, only replacing "&" and "/" characters is enough.

Sooo... now planning to just create a very minimalist version (some day), matching my minimalist JS skills. Dropping all that handy but too complicated code, and instead create a whole new, smaller script. Only for extracting the 2 download links to 320.mp4 + 720.mp4, plus title, statistics, description fields and the preview pic. With or without real json-functions, will see.

What I did wasn't complicated . I mean, yes, I have to make tests like you. Pure reverse engineering.

The complication is the original work itself. If it were by me, I'd make a simplistic script as well. I'm just keeping the original work and functionality as much as I can, as that debug you tell. That is not intended to debug. If I remember correctly by reverse engineering it, it was just to report to the original author by the script users an specific error message given by that debug strings.

The script is not intended for the mobile version. But I see it doesn't change much. Except that maybe your wish is too far. I only see the .3gp version for download (itag 17).

Also, the title appended to the URL does nothing. The file name doesn't depend on the URL, but on the filename served by the server, and usually it is videoplayback.xxx, except for 720p.



Edited 1 time(s). Last edit at 02/12/2020 04:02PM by JohnHell.