removing or ignoring line breaks for diff

[midwest_mike]

hello everyone! is there a simple way to remove line breaks/carriage returns from a text file (either with a vi command or with a linux command), or have diff automatically ignore them,
1 file has something like:
blah blah bla
h
& the other has
blah blah blah

& that's showing up as a difference - which for my purposes isn't really a difference. it's not just whitespace either. & just to be sure I'm already ignoring white space:
diff -B -i -b file1.txt file2.txt

it would be fine with me if each file was reduced to one long line.

thanks so much if someone happens to know this! take care, mike.

[midwest_mike]

if i removed all line breaks & reduced each file to one long line, that probalby wouldn't really help me, diff would just say "there's a differenc on line one".

hmmm... maybe there's nothing i can do. any thoughts?

[z1p]

Is there some 'rule' as to what is an ok difference? you may be able to do something with awk. Is there a way to determine when 2 lines should be combined into one?

[midwest_mike]

thanks for the reply, zip!
i'm not familar with awk but i see an online tutorial i'll read - hopefully that'll give me an idea of how to tackle this.

if you are interested, or curious, here's the problem - any quick thoughts or directions to point me in would be appreciated. but, not expected of course! it's my project, not yours.

a rule is: there are blank lines in the files. everything between two blank lines (or above/below a blank line - as i can check the 1st or last views manually) can be combined into one long line. as you can see, i'm trying to check to make sure the view code in two different databases is synchronized but the linebreaks are unpredictable (it's not a "long" or "pagesize" issue). the files are something i've spooled from user_views in each database, so:
file 1:
select * from whatever where some
thing

select * from whatever2 where something_else

file 2:
select * from whatever where something

select * from whatever2 where something_else

differences like those are plentiful & it's hard to see if any views are actually different.

[midwest_mike]

that's not a great way to get replies!

[kage]

Not sure if this is what you're asking, but to remove blank lines in VI, press 'dd' when the cursor is over the line you wish to be deleted. To join two lines together, use 'J' (shift+j).

[midwest_mike]

thanks kage,
that will help - w/ over 100 views, i hope it doesn't come to altering the .txt files manually. unless I can incorporate the 'J' with my awk file (if that' how awk works) i'll look into it.

[z1p]

You may be able to do it just with awk or you may be able to use sed (most likely along with awk) to get it done. It seems that the hardest part to script is determining what lines should be combined.

Combining lines with awk can be pretty simple. The following goes through the file and combines every 2 lines with each other. Changing the 2 changes the number of lines concatinated with each other. Coming up with the correct replacement for "NR%2 == 0" is the tough part.

awk '{prev=prev $0}; NR%2 == 0 {print prev;prev=""}' ~/tmp/junk

another way of writing the same thing is

awk '{prev=prev $0}; { if (NR%2 == 0) print prev, prev=""}' ~/tmp/junk

lastly this will combine any line that is exactly 20 characters long with the next line, all other lines are kept as is.

awk '{ if (length == 20) prev=$0; else print prev $0, prev=""}' ~/tmp/junk