A long time ago, I identified this failure but wasn't sure why it was
failing. I still don't know why, but the emergency fix at least prevents
other people from running into it. As a bonus, since we have a proper
logging facility now, we can actually log it as I desired to do back
then!
This commit tentatively fixes issue 729.
QDocument::toPlainText() has the stupid decision to convert nbsp to
spaces in it, which our users obviously hate. Unfortunately, this is
out of our control to fix completely. It is a very deep rabbit hole. :(
Typing non-breaking spaces in the editor now works. Reopening these
files at a later point has these characters remain intact.
What does NOT work is copy-pasting non-breaking spaces. These will end
up looking like normal spaces when you paste them somewhere else, be it
in Manuskript or some other document. In other words: it is impossible
for users to verify whether something is a non-breaking space or an
ordinary one.
I realize that it makes this partial fix meaningless for many. Sorry. :/
Partially fixes issue 738.
If a project only had one character, and it was marked as a POV
character, then that could lead to the accessing of a non-existent
second character. (Yes. That code was very broken.)
Fixes issues 843, 875, 896.
This takes care of the following warning in our log file:
Qt WebEngine seems to be initialized from a plugin. Please set
Qt::AA_ShareOpenGLContexts using QCoreApplication::setAttribute before
constructing QGuiApplication
In addition to fixing the bug, related code that allowed this one to
slip under the radar has been cleaned up. Validation of the FILENAME
argument is now performed during parsing.
* Changed default logging behaviour
We now log by default to a timestamped file in $datadir/logs/. No longer
shall restarting Manuskript after a crash wipe a very useful logfile.
Logs older than 35 days in the $datadir/logs/ directory are pruned
during startup. In case of subtle corruption detected a few weeks after
the fact, relevant logs might still exist to explain what had happened...
yet it does not come at the cost of infinitely gobbling up a users
storage space, either.
The --logfile (-L) argument can now utilize strftime() specifiers. A
special modifier %# is also supported which will insert the process id.
Besides being an added factor of uniqueness for a filename, it can also
be relevant to help identify the log file belonging to a misbehaving
Manuskript process.
* Added support-related items to Help menu
The 'Technical Support' item should lead to a landing page that will
guide the user to the most efficient way to resolve their problem.
How to report bugs and submit logs would be one of those.
The 'Locate Log File' item should open a file manager window with the
logfile of this session highlighted. Because Manuskript is still writing
to it, we first remind them of its limited use until Manuskript is
closed.
This approach was chosen because users might want to locate the file
prior to reproducing a bug, or because they'd like to look at other logs
from previous sessions.
* Updated translation files and added german translation
Co-authored-by: TheJackiMonster <thejackimonster@gmail.com>
* Fixed pandoc command arguments
`--base-header-level` throws a new export error because it is deprecated, now we should use `--shift-heading-level-by`
* Adjusted default and min value for base-header
Co-authored-by: Tobias Frisch <thejackimonster@gmail.com>
Due to my struggles reproducing the official build, I felt it might be
useful to log extra information regarding the version of PyInstaller.
Unfortunately, such information is not available due to the way things
work. However, during that process I came across some other interesting
details that would likely be useful when logged.
When implementing most of the logging code weeks ago, I got so caught up
in implementing and testing the Qt bits that I completely blanked on
actually implementing the Python-side of catching exceptions.
Unfortunately, it is not exactly trivial. PyQt complicates things, but
also Python itself isn't very helpful in older versions which will rob
the ability to properly log errors from threads. By the time I realized
we don't actually use the threading module and that catching the errors
on there does not actually help to fix the weirdness I was seeing, it
was already implemented. Thank you PyQt for surprising me yet again! :-)
I have tested this on Python 3.7 by raising random exceptions in various
parts of the codebase to make sure it behaves as expected. I haven't
quite gotten around to installing Python 3.8 yet but this gets it out
for testing. I'll hopefully not forget to do more tests in 3.8 when I do
the final squashing prior to a merge.
Some log messages use characters in the Unicode character set that
aren't typically represented in the older encodings. One such example
was the messages that happen when renaming an item and saving the
project.
Positive note though: errors while logging might not end up in the log
file when the log file is the cause, but they still showed on the
terminal with excessive detail. Also, despite the exceptions and errors,
the project I was testing successfully completed its saving procedures
as a subsequent reload showed. I am personally quite happy with the
degree of thought and care put into the Python logging system. :-)
During development, the version number does not have much meaning... but
when faced with a reported issue, you would still like to know in more
detail what version of the Manuskript code was at work there.
Knowing the exact git revision will hopefully make it easier to
troubleshoot such issues in the future.
Note: this code takes special care to not rely on external modules
(we have enough dependencies) or even the git executable (invoking a
program can be relatively slow on some operating systems). It might not
handle all the edge cases, but I think it should cover our needs well
enough.
Manuskript now logs the versions of modules and libraries powering them
for as far those are easily accessible. This includes all the optional
modules, too. None of this is visible on the terminal of course - unless
Manuskript is run with the --verbose flag. This clears up the last bit
of unnecessary console spam, leaving our users blissfully unaware.
Until we (and/or Qt) break something again, that is...
It had its own mini-logging facility to suppress the most useless debug
messages. Since the new facility does not show debug messages to the
user, but still allows them to end up in the log file, I replaced it
entirely with DEBUG-level logging.
Some spots have received an extra ERROR-level logging on top for the
sake of alerting the user to issues we would like to hear about, and to
avoid breaking up the hierarchical indenting done by the DEBUG-level
messages.
Some snippets have yet to be converted due to the more complex nature
of those snippets, and to keep things neat a separate commit makes more
sense for those.
Sometimes you want to do just one thing, and in the process of
implementing that one thing, you implement several others. This is one
of those types of commits.
Implementing the argparse library is for the sake of controlling the
logging behaviour as well as other future I have yet to implement.
It has all the standard goodies you'd expect, and I have also ported
over the existing commandline arguments. (They may need a bit of polish
still, but there is no regression compared to before, only improvement.)
The logger is because it really needed to happen for numerous reasons.
It still logs to the terminal, but by default it only does so for
messages classified WARNING and above. These are the things we actively
want users to see. But if this is not good enough, adding the --verbose
flag will increasingly show more (-v shows INFO level and -vv also shows
the DEBUG messages) so that us coders don't have to miss anything in the
most convenient location.
It also logs to a file with the very original filename manuskript.log. I
may have to add commandline and/or settings arguments to improve that at
some point in the future, but there distractions are endless.
The log file contains timestamps and module information for easy
interpretation that are not present on the terminal, and it contains all
messages classified DEBUG and up. Ideally users will just be able to
attach it to an issue(*) to deliver us all the information we need to
help them with their inquiry.
Last but not least, the other reason I needed logging implemented is
that Qt has its own logging framework, and I needed to figure out how to
siphon out the data and make it shut up. But there was no point in doing
that as long as our own logging facilities were lacking...
(*) I have yet to convert all existing print statements over to the new
system, but that is probably going to be the next commit. This one has
enough change in it already.
This is the quick way to patch this. I'd recommend changing the findFirstFile function in functions/__init__.py for a more permanent solution, but this should suffice for now.
When creating a new character, sets an appropriate importance level.
* If a character is selected, the new character has the same importance level.
* If a top-level importance level is selected, the new character has that level
* Otherwise, the importance level is zero
When i was using certain styles like cleanlooks or qt5ct-style, a TypeError was raising in cascade about the function not having enough arguments.
It looked like that, despite the last args of Qstyle.subElementRect() and Qstyle.sizeFromContents() were optional, it was still required to mention it (even if it was just None).
That TypeError was only appearing with certain styles, at startup or when changing styles in the settings window.
Two separate pull requests indicate an issue translating the warning
for when an import is attempted with PyQt/Qt versions 5.11 and 5.12.
As such change the warning message.
See PRs #668 and #701.
QtCore.Qt.GlobalColor does not have any accessors for the predefined
colors on PyQt versions before 5.11 despite the object itself existing.
It would have been nice if the documentation* had mentioned that object
being broken on older versions, but I should have tested with even older
versions of PyQt before submitting the original patch.
Apparently the most supported way to access these colors is through the
Qt namespace itself, but those aren't documented in the slightest. Ugh.
My apologies to all those affected. Fixes issue #659.
*: https://www.riverbankcomputing.com/static/Docs/PyQt5/api/qtcore/qt.html#GlobalColor
A previous fix (5f9ea3) inadvertently broke the progress bar by
converting to the wrong data type. (See issue #561 / PR #609).
While checking the code I realized the problem occurred primarily
because we weren't checking the validity of the values closer to the
source. Doing so alleviates the need to check elsewhere.
In the hope of inspiring a more systematic approach, a new uiParse()
utility function has been added to curb the further growth of toXxx()
functions that exist solely to validate user input.
There is no doubt room for improvement, both on the end of the new
uiParse() function as well as the spot where it is used. Ideally, the
data that comes out of the model should already be 'safe', but since
this is a bugfix for a bugfix I want to keep waves to a minimum.
This commit fixes issue #652.
This commit sets the "keep revisions" setting to disabled by default
for new projects. It also removes the test cases associated with
revisions.
The revisions feature as currently implemented is of questionable use.
It has been a source of performance issues for users, and has consumed
developer time in attempts to address its shortcomings. This commit
is the first step to deprecate the current revisions feature.
See PR #651
This commit restores the functionality that prevents spell checking a
word that is being actively typed at the end of a paragraph.
The goals for the spell check word match regexp are:
A. Words should include those with an apostrophe
*E.g., can't*
B. Words should exclude underscore
*E.g., hello_world is two words*
C. Words in other languages should be recognized
*E.g., French word familiarisé*
D. Spell check should include word at absolute end of line with no
trailing space or punctuation
*E.g., tezt*
E. Spell check should ignore partial words in progress (user typing)
*E.g., paragr while midway through typing paragraph*
This commit addresses all five of the above goals.
HISTORY:
- See issue #166 and commit 6ec0c19 in the 0.5.0 release.
- See issue #283 and commit 63b471e in the 0.7.0 release.
Also fix minor incorrect utf-8 encoding at top of source file.
See issue #611 and pull request #642.
The warning previously added in PR #612 for Manuskript users with Qt
5.11 / 5.12 has shown itself, in my opinion, to be overly annoying.
This is because the warning *always* displays on systems with the
affected Qt versions even though the crash is verified to happen with
the import feature only.
Additionally the process to upgrade to a newer version of PyQt / Qt is
not trivial for users who rely on pre-built packages and do not run
from source code.
Because the crash has been verified with the Import feature only, limit
the scope of the warning to the Import feature.
The user can configure a language for Manuskript in the dialog, but
before that setting is ever written to disk, there is the default
behaviour that tries to auto-detect the best language to show based on
the configuration of the device it is running on.
While doing my due diligence on issue #619, I realized we were relying
on the system locale, which is not necessarily equivalent to the
language the user is working with. Worse still: a user can have multiple
preferred languages for their user interface, and our old approach might
actually offer them the 'wrong' language. This patch fixes this.
It also refactors and comments things a little bit where necessary.
Issue #619 revealed an unintentional overlap between the auto-detection
of the locale and the implicit builtin English language, which resulted
in users being unable to select the builtin English language when their
device was configured with a locale that we happen to have a translation
for. The code has been rewritten to more clearly separate auto-detection
and the final fallback that is the builtin English translations.
This code change sets:
- Fequency Analyzer tool default first tab of "Word frequency"
(was Phrase frequency)
Steps to set default window tab:
1. Start Qt Designer
2. Open .ui file
3. Ensure that each selected window tab is the one desired as default
4. Save .ui file
5. Exit Qt Designer
6. Generate .py file with: make ui
See PR #623
This code change sets:
- Character pane default first tab of "Basic info" (was Notes)
- Character pane Basic info "Name" as the default first field (was Motivation)
- Plots pane default first tab of "Basic info" (was Resolution steps)
Steps to edit tab order and default window tab:
1. Start Qt Designer
2. Open .ui file
3. Choose menu **Edit -> Edit Tab Order**
4. Ctrl-click on item just before the first incorrect tab order item
5. Click other items in order until remaining order is correct
6. Ensure that each selected window tab is the one desired as default
7. Save .ui file
8. Exit Qt Designer
9. Generate .py file with: make ui
See https://doc.qt.io/qt-5/designer-tab-order.html
This code change implements a portion of issue #244
Last time I touched this code, I went in looking for a specific problem,
and came out with a fix specific to that issue. That fix was not wrong,
yet it hardly covered all the problems present in the code once one took
into account issues like:
- local vs remote resources,
- relative vs absolute paths,
- different operating systems behaving differently, and
- Qt being uniquely buggy on different platforms.
The major part of it was fixed by using QUrl.fromUserInput(), which does
the exact kind of auto-detection for the nature of the resource that we
were in need of.
The rest of the issues were fixed by creating a number of test cases and
fixing problems as they popped up. Testing was done in Windows & Ubunty
against the above-mentioned test cases, which can be found in PR #629.
Regarding ImageTooltip.supportedSchemes
When QUrl.fromUserInput() misidentifies the scheme on Linux, it causes
all resemblance between the original request and the reply.request() in
the finished() signal to be lost, which results in this item getting
stuck in the ImageTooltip processing pipeline.
Limiting the supported schemes to the ones most commonly encountered
('file', 'http', 'https' and the schema-less local paths) is the only
reliable method I have found to work around this particular bug in Qt.
Windows 10 has supported a 'dark theme' option for a while, and the fact
Manuskript is like looking into a bastion of bright white while using it
is bothersome to say the least.
Since this is a setting defined as the OS level, I believe this should
be something Manuskript automatically adjusts itself to match, thus the
lack of a configurable setting on the Manuskript end.
Qt has garbage documentation. I dug into Qt 5.0 source code to make sure
the crashing call can be safely left out for older versions, and that is
indeed the case. setSupportedSchemes() appears to be introduced together
with the future to select non-local files in the dialog, but we only
cared about local files to begin with.
I happened to notice the UI briefly locking up when testing my previous
changes on some of the Pandoc exporters. While the bigger mess I found
along the way is more than a little fix can handle, this stopgap measure
will at least stop us from running pandoc when it isn't needed.
According to issue #608 we were silently overwriting files when there
was a suffix being generated for a chosen export filename when one was
missing to begin with. Unfortunately, this helpful feature avoids all
the conveniences offered by QFileDialog in regards to alerting the user
to overwriting an existing file. Worse still, this feature already
exists in QFileDialog and the native APIs it can rely on.
This patch reimplements QFileDialog.getSaveFileName to allow the use of
the default suffix feature as functions.getSaveFileNameWithSuffix and
removes most of the magic involved with the old solution.
It is the only FileDialog in the entire codebase that does not conform
to the rest of the OS like its brethren, and it stuck out like a sore
thumb because of it.
Some bugs are out of our reach to fix, but can still impact the user
considerably. Because losing progress always hurts, we want to make
the user aware of the risks before any tears are shed. (PR #612)
See PR #615
This rename is being done to clarify that when a user enables "Save on
project close" then the project will be saved whenever the user
chooses to close the project or to quit Manuskript.
Note that the actual name of the setting saveOnQuit should also be
changed but instead has been marked as a future TODO because it
involves a change in the project file format.
See PR #615
The Travis CI tests began failing after merging Pull Request #583.
Log snippet:
----------------------------------------------------------------------
...
Ref not implemented
PASSED
manuskript/tests/ui/test_welcome.py::test_autoLoad QXcbConnection: XCB error: 8 (BadMatch), sequence: 613, resource id: 2097162, major code: 42 (SetInputFocus), minor code: 0
QXcbConnection: XCB error: 8 (BadMatch), sequence: 619, resource id: 2097168, major code: 42 (SetInputFocus), minor code: 0
QXcbConnection: XCB error: 8 (BadMatch), sequence: 625, resource id: 2097171, major code: 42 (SetInputFocus), minor code: 0
----------------------------------------------------------------------
When running "pytest -vs" locally, which is a command used in our
.travis.yml file, a dialog to "Save project?" is displayed. Because
the test scripts use the "saveOnQuit" default setting of *enabled*,
the "Save project?" dialog should not be displayed.
In other words when a call is made to close the project, a "Save
project?" dialog is incorrectly displayed because the dirtyProject
flag is set, but so too is saveOnQuit set to True. What should happen
is an automatic save with no prompt. This PR fixes this logic so that
the Travis CI test suite completes successfully.
Turning off the timer for saveTimerNoChanges just like the code
already did in closeProject() for saveTimer fixes this bug. Easy.
But how to prevent this kind of race condition in the future?
Several related routines have been adjusted to fail gracefully or report
a bug to the console when something goes wrong, depending on what is
most suitable for that bit of logic.
Intending to learn more about the way Manuskript goes about saving the
project in order to figure out how to tackle some recent saving-related
issues, I stumbled into learning that Manuskript likes to save data a
whole lot. Too much, in fact. When I close the project with unsaved
changes, I expected those changes to not be saved... but they were. This
completely subverts my expectations of a program using typical
file-based operations involving opening, saving and closing files.
There are three more settings that influence when the program saves, and
I personally consider them a bit overkill or even detrimental to the
stated purpose. What if Manuskript forces a save when nothing was
changed and something goes wrong? Saving too much can in fact be
dangerous!
For now, I have left existing functionality as-is, but I would prefer to
respect the dirty flag I have introduced in this commit for at least the
'save-on-quit' and 'save every X minutes' features. (The third is
smarter and only triggers after noticing changes, so it is less
important.)
Making sure the dirty flag works as expected is the first step in making
such changes in the future.
UI-wise, this commit now offers the user the opportunity to save their
changes, discard them, or outright cancel their action entirely when
performing a destructive action on a dirty project. As of this commit, I
have identified two of such scenarios:
1) closing the project,
2) closing the window with save-on-quit turned off.
If I missed any, do let me know. But for now, maybe now I can finally
start digging into those issues that sent me down this rabbit hole...
See issue #561.
The problem appears to be a due to a combination of factors, such as:
- Python does not automatically convert an empty/blank variable to the
integer zero (0)
- Default goal value is empty/blank for a new Text (scene)
- Asynchronous events can occur such that the change in the Outline
pane of a new Text (scene) goal from empty/blank to a value is not
saved to the data model prior to the update event in the Editor pane
accessing the model value for the word count progress display.
Steps to Reproduce:
1. Start manuskript and create new project (no template).
2. Select **Outline** pane.
3. Click "Text Plus" icon to create a text (default name "New")
4. Select **Editor** pane.
5. Click on **New** to display empty text.
6. Select **Outline** pane.
7. Double-click the empty area on **New** line under title **Goal**,
type in "300", and press **Enter**.
Note that manuskript crashes with a segmentation fault.
Work around the crash by using the already existing manuskript
function toInt() which handles conversion of empty/blank values to
integer value zero (0).
If an invalid character is inserted into the text, such as a "^L" (ASCII 0x0C)
when copy-pasting from a google document that has a page break in it, a crash
will happen as the character cannot be inserted into XML. This patch removes
those invalid characters from the text so the revisions.xml can be saved.
Fixes#562
The About/Settings/Import/Export/ExportManager windows were all created
in odd places, usually to the left of the main window, which meant outside the
desktop area with little overlap if the main window is maximized. The logic in
centering the window on its parent was wrong. This fixes it.
Describing all the rabbitholes that I and kakaroto have gone through
while debugging this one until dawn can frankly not do enough justice to
the crazy amount of rubberducking that went on while trying to fix this.
This bug would be triggered whenever you had a document open in the
editor and then moved an ancestor object downwards (visually) in the tree.
Or when you simply deleted the ancestor. Depending on the exact method
that caused the opened item to be removed from the internal model, the
exact nature of the bug would vary, which means this commit fixes a few
different bits of code that lead to what appears to be the same bug.
In order of appearance, the bugs that ruined our sleep were:
1) The editor widget was trying to handle the removed item at too late a
stage.
2) The editor widget tried to fix its view after a move by searching for
the new item with the same ID, but in the case of moving an object down
it came across its own old item, ruining the attempt.
3) The editor widget did not properly account for the hierarchical
nature of the model.
Upon fixing these the next day, it was revealed that:
4) The outlineItem.updateWordCount(emit=False) flag is broken. This
function would call setData() in several spots which would still cause
emits to bubble through the system despite emit=False, and we simply got
lucky that it stopped enough of them until now.
This last one was caused by a small mistake in the fixes for the first
three bugs, but it has led to a couple of extra changes to make any
future bug hunts slightly less arduous and frustrating:
a) When calling item.removeChild(c), it now resets the associated parent
and model to mirror item.insertChild(c). This has also led to an extra
check in model.parent() to check for its validity.
b) The outlineItem.updateWordCount(emit=) flag has been removed entirely
and it now emits away with reckless abandon. I have been unable to
reproduce the crashes the code warned about, so I consider this a code
quality fix to prevent mysterious future issues where things sometimes
do not properly update right.
Worthy of note is that the original code clearly showed the intention to
close tabs for items that were removed. Reworking the editor to support
closing a tab is unfortunately way out of scope, so this intention was
left in and the new fix was structured to make it trivial to implement
such a change when the time comes. An existing FIXME regarding unrelated
buggy editor behaviour was left in, too.
Many thanks to Kakaroto for burning the midnight oil with me to get to
the bottom of this. (I learned a lot that night!)
Issues #479, #516 and #559 are fixed by this commit. And maybe some others,
too.
Add support for 6.3.8 which has delete_dictionary_entry and do not use gzipped
pickle. Also give higher priority to symspellpy vs pyspellchecker.
List symspellpy dictionaries by order of cached vs non-cached.
symspellpy 6.3.8 is now the minimum version required and add support for showing
that information to the user.
Also add support for spellcheck libraries that are installed but without dicts.
SymSpell is a great spellchecker which works a lot faster than
pyspellchecker for finding suggestions but is a bit slow at
loading dictionaries (about 15 seconds initially, 2 seconds if
using a cached version).
SymSpell also doesn't come with dictionaries, so the code is currently
using dictionaries from pyspellchecker, so if pyspellchecker isn't
installed, then the user won't see any available dictionaries.
Eventually, would need to have an interface for people to manage
dictionaries for it.
Improves the custom dictionary support by making it more generic
and moving it to the base class. Also makes PyEnchant uses a custom
PWL (Personal Word List) file within manuskript's resources directory
and made pyspellchecker detect available languages automatically.
This modifies the Spellchecker abstraction to add a new dictionary support, with
support for pyspellchecker. It also changes the main UI so that multiple libraries
can be supported and dictionaries provided to the user. The custom dictionary of
pyspellchecker has to be handled manually, and the performance and words of this
library isn't on par with PyEnchant, but at least it works with 64 bits.
Fixes#505
This is in preparation for adding support for additional spellchecking libraries
other than PyEnchant which seems to be unmaintained and does not build in
Windows 64 bit.
Issue #549 was caused because the request and reply object urls are not
guaranteed to be the same. Redirects are the most common cause, but a
malformed URL apparently also qualifies. We now make sure to look at the
original request.
Because the code confused me while I was working on it, I decided to
refactor and document it in order to understand what was going on. I am
glad I did: I found another crashing bug involving the rapid-firing of
tooltip requests, and the processing dict never had its entries removed
either, leading to a (very slow) memory leak over time.
All is good in the world of image tooltips now.
The 'displays' system of panels is now changed into simply a settings system
where settings can be associated to widgets. The new API is :
addWidgetSetting, addSetting and setSettingCallback.
The top panel was reworked to have the settings appear in the order of the
widgets, and the path/title choice was changed into a Title widget with a
"Title: Show Full Path" setting.
Realizing that the show/hide progress was being ignored if we navigate to a
scene without a goal set. Also, if we go fullscreen on a scene without a goal
then navigate to a scene with one, the progress wouldn't get shown. Adding the
"Auto Show/Hide" setting fixes the issue with all use cases.
This makes the fullscreen editor much more powerful in terms of navigating
through chapters and scenes. This should make issue #234 users happy and
fix#444.
Top panel now has left/right arrows to navigate through the scenes. It will
automatically find the next/previous text item and display it, navigating through
the outline tree.
There's also a "New document" icon which will create a new text entry immediately
after the current one and switch to it.
Navigation can also be done using Alt+Page-Up, Alt+Page-Down or Alt+Left and Alt+right
shortcuts (Fixing #444).
There's now also the option between Title or Path for the top panel, if Path is chosen
then the full path of the scene is displayed and clicking on the scene or parent items
opens a menu to quickly switch to the selected chapter/scene. Selecting a folder will
automatically display the first text entry available in that folder.
